Fu lly D ist r i buted Representati on

Pentti Kanerva
RWCP Neuro SICS Laboratory
Real World Computing Partnership
Swedish Institute of Compute Science
e-mail: kanerva@[Link]

Abstract
A fully distributed representation based on the binary spatter code is described. It is shown how the
information of a conventional record with fields is encoded into a long random bit string, or a holistic
record, that has no fields, and how the fields are extracted from the holistic record. It is argued that
holistic representation should be used in modeling high-level mental functions.

1. INTRODUCTION representation it is pointed out that the present system

is related to Plate’s (1994) Holographic Reduced
Local representation—records with fields—so domi-
Representation and is described in the same terms.
nates our computing practices that we are hardly
aware of alternative representations, or even the need 2. RECORDS WITH FIELDS
for any. By representation is meant, simply, how
Local representation stores information in records
information is laid out in bits in some physical
with fields. Figure 1 shows some examples. A binary
medium, for example in a computer memory or a
attribute- or feature-vector is the simplest example.
neural net. With local representation, the meaning of
Such a vector for N attributes is then a record with N
a bit—what a bit refers to—is tied to its location. To
one-bit fields. Figure 1a shows unary encoding of the
decide what a bit pattern is, we must therefore know
English alphabet with such vectors. Unary encoding
where it comes from. This creates the need for a sys-
and its real-valued counterpart are common in statis-
tem that keeps track of where information is located.
tical classifiers (e.g., artificial neural nets), with each
A computer program is such a system.
class having its own output variable. Such codes are
The search for distributed representations (e.g.,
easy for us to interpret but their use of bits is waste-
Anderson, 1995; Hinton, 1990; Hinton et al., 1986;
ful.
Pollack, 1990; Smolensky, 1990; Touretzky & Geva,
Figure 1b shows a record for a name made of 12
1987; Touretzky, 1990) has grown out of the realiza-
five-bit fields for the letters. The representation of a
tion that traditional representation, as used in com-
letter within a field is now distributed, as no particu-
puters, is not only artificial but that it may actually
lar bit will tell whether the letter is an A, for example.
hinder the development of the kind of computing that
In Figure 1c such a record is used as the name field of
makes brains intelligent. One reason would be that
another record that has fields also for sex and age.
traditional representation relies on a program to inter-
pret it, and that intelligent programs of the traditional
kind are extremely difficult to find with automated
(a) A B C D E F G . . . . X Y Z
learning methods because programs are unstable. A
superficially minor change—an incorrect bit or a E 0 0 0 0 1 0 0 . . . . 0 0 0
1 2 3 . . . . . . . . . . . . . . . . 26
misplaced symbol—can wreck an otherwise good
program. Learning methods that rely on incremental (b)
improvements cope poorly with instability, and good P A T space ...
programs are so rare among all possible programs PAT 10000 00001 10100 00000 0 . . . . 0
that finding them by chance is hopeless. 1 . . 5 6 . .10 11...15 16...20 21 . . . 60
Written as a tutorial, this paper demonstrates with (c)
simple examples a fully distributed alternative to name sex age
local representation, so as to make the ideas accessi- PSmith PAT _ _ _ _ _ _ _ _ _ male 66
ble. Properties of the representation are discussed, 1 . . . . . . . . . . . . . . . 60 61 62 . . . . 68
and pointers to past and future research are given. For
the benefit of those already familiar with distributed Figure 1. Local representation: records with fields.

358
 Figure 2 makes the point that two records that dif- composed record also has N bits, with 0s and 1s
fer from each other minimally can mean very differ- equally probable, and thus it can, in turn, be assigned
ent things. It demonstrates that traditional data as a value to a field (i.e., bound to a variable) and
representation is unstable in the same sense as tradi- chunked into further N-bit records. This property
tional programs are. Figure 3 shows that the same makes the code recursive.
information can take very different forms. The majority rule can be realized as a thresholded
sum: by thresholding the columnwise sums at half the
3. DISTRIBUTING THE RECORD number of chunked vectors. The holistic record for
In this section we see how the information of a tradi- Pat Smith—the encoding of ‘name = Pat & sex =
tional record, the three-field record of Figure 1c, can male & age = 66’—can then be expressed as
be encoded into a binary vector without fields. Each
PSmith = [name⊗Pat + sex⊗male + age⊗66]
field will be distributed fully, over the entire vector,
so that each bit contains some information about where the brackets […] indicate thresholding. The
every field. The resulting record is called holo- vectors that are combined by chunking are also called
graphic or holistic. parts. To break ties when the number of parts is even,
Codewords or -vectors. The holistic record is built we can chunk a random vector R with them (that pos-
of codewords for field names (variables, roles) and sibly is a function of the parts).
for values that occupy the fields (fillers). All code- Chunking can be compared to creating a pointer to
words are long, random bit strings or binary N-vec- a conventional record (see Fig. 3), which then repre-
tors; N = 10,000 will be used in our examples. The N sents the record. This is how symbolic (list) struc-
bit positions or coordinates or columns (as in a table tures are constructed. The main similarity between
where the vectors are the rows) of a codeword are the two is uniformity: all chunked records have the
independent of each other, and 0s and 1s are equally same number of bits, N, and all pointers have the
probable, p = Pr{1} = 0.5. To encode the record for same number of bits, L. One major difference is that
Pat Smith, one such codeword is needed for each of N is much larger than L (usually L < 30), and the
‘name’, ‘Pat’, ‘sex’, ‘male’, ‘age’, and ‘66’. The other, which is related to the first, is that a conven-
codewords are written in boldface: name, Pat, sex, tional pointer hides its relation to the contents of the
male, age, and 66. record, whereas chunking leaves the parts visible.
Binding and chunking. A holistic record is com- Visibility of Parts. When K codewords are
posed in two steps called binding and chunking. chunked, the resulting codeword bears more than
Binding encodes a field, and chunking combines the chance resemblance to its constituent parts. The
fields into a record. Binding is done with coordinate- smaller K is, the greater the similarity. If K = 3 and if
wise Boolean Exclusive-OR (XOR, ⊗), so that ‘name A, B, and C are random and independent and X = [A
= Pat’ is represented by the 10,000-bit codeword + B + C], how close is X to A, or PSmith to
name⊗Pat. This corresponds to storing ‘Pat’ in the name⊗Pat? The expected (Hamming) distance, rela-
name field of a traditional record. Similarly, sex⊗ tive to N, is the probability that the corresponding
male and age⊗66 encode the other two fields. bits of X and A differ, δ(X, A) = Pr{X n ≠ A n}. If a bit
The encoded fields are chunked into a holistic
record according to the majority rule. Each bit of the
name sex age
record will be a 0 or a 1 according to which of them
appears more often in that position, or column, in the PSmith PAT _ _ _ _ _ _ _ _ _ male 66
1 . . . . . . . . . . . . . . . 60 61 62 . . . . 68
(three) vectors that are being chunked. When the
number of chunked vectors is even and there are ties,
sex age name
they are broken at random with probability 1/2. The
PSmith 2 male 66 PAT _ _ _ _ _ _ _ _ _
1 2 . . . . . 8 9 . . . . . . . . . . . . . . 68

name sex age

PSmith PAT _ _ _ _ _ _ _ _ _ male 66
Pointer to PSmith,
PSmith F PAT _ _ _ _ _ _ _ _ _ female 66 1 . . . . . . . . . . 32 a memory address
1 . . . . . . . . . . . . . . . 60 61 62 . . . . 68

Figure 2. Two very different Pat Smiths, two very Figure 3. Three representations of PSmith, three
similar bit strings. very different bit strings.

359
of A is a 0 and the same bit of X is a 1, it must be a 1 between a chunk and its parts is low, it is very signif-
also in B and C to constitute a majority. This happens icant when the code is wide (N > 1,000) and the parts
with probability 0.25, and it is the same when a bit of are few (K < 15).
A is a 1. So the bits of X and A differ with probability Decoding. When XOR is used for binding, as in
0.25, and thus X is much closer to A (and to B and to name⊗Pat, it is used also for “unbinding” or decod-
C), and PSmith to name⊗Pat, than what it would be ing. To extract Pat from an encoded name-field, we
by chance: d(X, A) = 0.25 ± 0.0043, whereas d(A, B) XOR with name:
= 0.5 ± 0.005 by chance (getting d ≤ 0.25 by chance
name⊗(name⊗Pat) = (name⊗name)⊗Pat = Pat
is as unlikely as getting no more than 2,500 heads in
10,000 tosses of a coin). The standard deviation of because XOR is associative and is its own inverse
distance is based on the binomial distribution for N = function. This is called probing.
10,000 and δ = 0.25, 0.5 and it is given by σ = Probing with name is used also for extracting Pat
(δ(1 – δ)) ⁄ N . from the chunked record PSmith. Because PSmith
For K ≠ 3 the distance is the following. If K is resembles name⊗Pat, name⊗PSmith resembles
even, we first add in a random vector R for a tie- name⊗(name⊗Pat), which is Pat. In fact, the dis-
breaker and write it as X = [A + B + … + D (+ R)]. tance d(name⊗PSmith, Pat) = d(PSmith, name⊗
Now K is odd, and the expected distance is Pat) = 0.25 ± 0.0043, because XORing with the same
vector, name, leaves the distances between vectors
δ(X, A) = --- −  K – 1  2K
1
2 /
(K – 1) ⁄ 2
unchanged. In the following we will write name⊗
PSmith as Pat′.
which is approximately 0.5 − 0.4/ K – 0.44 (this Clean-up Memory. We can think of Pat′ as a
approximation is an improvement over one based on noisy version of Pat that needs cleaning up. For that
Stirling’s factorial formula when K = 3, 5, 7, …, 49). purpose the system has a clean-up memory that keeps
The expected distance is shown in Figure 4. The track of all valid codewords—ones that have been
thing to notice is that d need not be far from 1/2 to be defined so far—and that takes noisy codewords as
significantly different from it when N is large since inputs and produces noise-free codewords as outputs.
its standard deviation is so close to zero. It is in this (The valid codewords mentioned up to now are
statistical sense that a chunk resembles its parts and name, Pat, sex, male, age, 66, and PSmith, whereas
differs from unrelated codewords. name⊗Pat, sex⊗male, and age⊗66 need not be
The distance δ is related to the correlation coeffi- stored in the clean-up memory.)
cient (normalized covariance) by ρ = 1 − 2δ (the rela- We still need to assure that the decoding is unam-
tion is linear also when p ≠ 0.5, although it is more biguous. Although Pat′ is close to Pat, it may also be
complicated). The right margin of Figure 4 is labeled close to other valid codewords in the clean-up mem-
with these correlations. Although the correlation ory. So let us look closely at the structure of Pat′:
Pat′ = name⊗PSmith
= name⊗[name⊗Pat + sex⊗male + age⊗66]
0.5 0
which equals
0.4 0.2 [name⊗(name⊗Pat) + name⊗(sex⊗male) +
CO RRE LA TIO N, ρ

name⊗(age⊗66)]
DI STA N CE, δ

0.3 0.4
In other words, the XOR distributes over the chunked
0.2 0.6 parts.
To see why a⊗X = a⊗[A + … + D (+ R)] = [a⊗A
0.1 0.8 + … + a⊗D (+ a⊗R)], first consider the places (i.e.,
bit positions or columns) where a has a 0; in those
0 1 places XORing with a has no effect on X or on any of
1 3 5 7 9 11 13 its parts A, …, D, or R, so that in those places distrib-
SET SIZE, K utivity holds trivially. Secondly, wherever a has a 1 it
flips the bits of X, and we get the same result by flip-
Figure 4. Distance and correlation between a holistic ping those same bits in A, …, D, and R and then
record and its parts as a function of the number of applying the majority rule—wherever 0s were in the
parts (set size) K. majority now 1s are, and vice versa.

360
 The above expression gives us therefore they are not in the clean-up memory. They
act merely as random noise (their distance to valid
Pat′ = [Pat + name⊗sex⊗male + name⊗age⊗66]
codewords is 0.5 ± 0.005).
which shows that, due to its structure, Pat′ resembles Figure 5 summarizes holistic encoding of a con-
also the vectors name⊗sex⊗male and name⊗age⊗ ventional record of Figure 1c and extracting a “field”
66 (δ = 0.25). But these are not valid codewords and from the holistic record.

name 00111…11

Pat 10010…01
⊗ 10101…10 name = Pat

sex 10001…10

male 11111…00
⊗ 01110…10 sex = male

age 10010…00

66 01110…00
⊗ 11100…00 age = 66

22311…20 (+)

> 3/2

11100…10 PSmith (A)

≈ Pat 11011…01 ⊗ 00111…11 name

CLEAN-UP
MEMORY

Pat 10010…01 (B)

Figure 5. Holistic encoding and decoding. ‘name = Pat & sex = male & age = 66’ has been composed into the
holistic record PSmith (A), and then the name has been extracted from it (B). Codewords are random 10,000-
bit strings, ⊗ is bitwise Boolean XOR, and bitwise sums (+) are thresholded at 3/2, constituting the majority
rule. ‘≈ Pat’ means that the result is approximate but close enough for the Clean-up Memory to identify it.

361
 4. PROCESSING EXAMPLES Table 1
The following examples provide insight into holistic Four Records for Holistic Encoding
representation. Let us assume that we have four
records with identical structure, as shown in Table 1. name sex age
They give rise to 13 codewords in the clean-up mem- PF3 Pat female 33
ory, namely, name, Pat, Lee, sex, male, female, age, PM6 Pat male 66
33, 66, PF3 = [name⊗Pat + sex⊗female + age⊗33], LF6 Lee female 66
PM6 = [name⊗Pat + sex⊗male + age⊗66], LF6 = LM3 Lee male 33
[name⊗Lee + sex⊗female + age⊗66], and LM3 =
[name⊗Lee + sex⊗male + age⊗33].
We now ask the following question: “What is the
age of Lee who is a female?” This refers to LF6, and which, by distributivity, equals
we look for 66. [PM6⊗Pat⊗Lee + PM6⊗male⊗female + PM6⊗R]
Case 1. In this case we assume that the codewords
name, Lee, sex, female, and age are known to us, but Substituting [name⊗Pat + sex⊗male + age⊗66] for
LF6 is unknown even if it is in the clean-up memory. PM6 and applying distributivity a second time gives
Solution 1. We can first approximate LF6 by PM6⊗T = [[name⊗Pat⊗Pat⊗Lee
replacing the unknown age⊗66 with a random vector + sex⊗male⊗Pat⊗Lee
R, giving LFx = [name⊗Lee + sex⊗female + R]. It + age⊗66⊗Pat⊗Lee]
contains 25% noise relative LF6, 37.5% relative to + [name⊗Pat⊗male⊗female
PF3 and to LM3, and 50% relative to the other ten + sex⊗male⊗male⊗female
codewords. The distances with standard deviations + age⊗66⊗male⊗female]
for N = 10,000 are d(LFx, LF6) = 0.25 ± 0.0043; + PM6⊗R]
d(LFx, PF3), d(LFx, LM3) = 0.375 ± 0.0048; and
d(LFx, other) = 0.5 ± 0.005. Because the standard which simplifies to
deviations are so small, cleaning up LFx will find the PM6⊗T = [[name⊗Lee + R1 + R2]
correct LF6 with very high probability. Thereafter + [R3 + sex⊗female + R4] + R5]
we can probe LF6 with age and clean the result to 66.
We will write that as age⊗LF6 → clean-up (0.25) → where the R i stand for vectors that act as random
66, where the 0.25 means that the amount of noise to noise, as nothing corresponding to them is stored in
be cleaned up is 25%. the clean-up memory. From this last expression for
Case 2. In this case Pat, male, PM6, Lee, female, PM6⊗T we see why mapping with T takes PM6
and age are known, but name, sex, and LF6 are near LF6: two of its parts are the same as in LF6,
unknown. with some noise added. The breakdown of LFx in
Solution 2c (c for ‘conventional’). We can find Solution 1 is similar except that the two known parts
name and sex by Pat⊗PM6 → clean-up (0.25) → there are noise-free. In that case the result is 25%
name and male⊗PM6 → clean-up (0.25) → sex. noisy compared to the present 37.5%.
From here on we can apply Solution 1. Case 3. Now 33, PF3, and LF6 are known. The
Solution 2u (u for ‘unconventional’). Here we use codewords 33 and PF3 would allow us to find age,
the correspondences Pat ↔ Lee and male ↔ female and with it we could probe LF6, but it is instructive
to form the mapping T = [Pat⊗Lee + male⊗female to look at a less conventional one-step solution.
+ R] and with it map PM6 as a whole: PM6⊗T → Solution 3a (a for ‘analogous’ [and also for
clean-up (0.375) → LF6. The other two distances ‘ambiguous’]). Think of a 33-year old Pat (PF3) ask-
that differ from 0.5 are significantly greater than ing Lee (LF6) the following question: “I’m 33; what
0.375, namely, d(PM6⊗T, PF3), d(PM6⊗T, LM3) are you?” Of course Lee could answer “I’m female,”
= 0.44 ± 0.005. Finally, age⊗LF6 → clean-up (0.25) but that would seem unnatural whereas “I’m 66”
→ 66, as above. seems natural. We will see how the holistic represen-
Solution 2u provides an opportunity to look at the tation favors the latter answer over the former.
inner workings of holistic representation, its underly- Since A = 33⊗PF3 ≈ age and age⊗LF6 ≈ 66, we
ing algebra. PM6⊗T is first written out as can try A⊗LF6 directly without first recovering age.
We get that A⊗LF6 is equally far from 66 and 33—
PM6⊗T = PM6⊗[Pat⊗Lee + male⊗female + R] d(A⊗LF6, 66), d(A⊗LF6, 33) = 0.375 ± 0.0048—

362
and it is 0.5 away from all other valid codewords. So specific location has a specific meaning; it causes a
although we do not find 66 unambiguously, a true but certain muscle fiber to contract, for example. How-
anomalous answer such as “I’m female” is not even ever, the further away from the periphery a neural
suggested. The expression (33⊗PF3)⊗LF6 is easily circuit is, the less specific its individual neurons are.
broken down algebraically, in the style of Solution Our present understanding is that higher mental func-
2u, and the breakdown shows how the two equally tions such as abstract thought and language depend
likely answers emerge. heavily on distributed representations.
Comments on the Examples. The above examples This paper demonstrates holistic distributed repre-
give cause for several comments. sentation with the binary Spatter Code (Kanerva,
1. The four records in Table 1 were chosen so that 1994, 1995, 1996), which is a particularly simple
none of them is determined by one attribute alone, so form of Holographic Reduced Representation (HRR;
although the records are few, they are nontrivial. Plate, 1994). The HRRs studied by Plate use real and
2. The records need not have identical structure. complex vectors, but all three—real, complex, and
For example, records that do not encode age are auto- binary—are related to each other mathematically.
matically ignored, and records that encode all three Plate’s thesis includes a comprehensive survey of the
attributes plus additional ones would give somewhat work leading to HRR and is highly recommended.
noisier results but would not confuse matters other- Holistic representation is uniform in the following
wise. Also, the codewords used in encoding the sense: all things—objects, properties, relations,
records can be noisy; there merely will be more noise attributes, values, composed structures, mappings
to clean up. Notice, also, that the bit pattern of a con- between structures—are random points of an enor-
ventional record depends on the order in which the mous vector space: its dimensionality N > 1,000. The
fields appear (cf. Fig. 3), whereas the order becomes vector components (the columns) are independent
irrelevant when the information is encoded into a and identically distributed (i.i.d.). In the real HRR
holistic record. they are distributed normally with zero mean and 1/N
3. The examples are not meant to suggest that tra- variance, in the complex HRR they are distributed
ditional data bases should be encoded in this holistic uniformly over the unit circle (magnitude = 1), and in
manner. These examples work because the records the binary HRR (i.e., the spatter code) 0s and 1s are
have few fields. If the fields are too many, extracting equally probable. Total absence of fields sets holistic
them from a holistic record is unreliable. However, representation apart from traditional representation.
holistic records could supplement traditional data Holistic representation makes it possible to com-
bases, but how that should be done remains to be bine structure and content, or syntax and semantics,
researched. so that the distance between codewords δ, or their
4. Solutions 2u and 3a point to a fundamentally correlation ρ, reflects similarity of both structure and
new way of computing, namely, by holistic mapping, content: it reflects the similarity of meaning. That
which is a mapping between points of a very-high- makes holistic representation an interesting candidate
dimensional space. More is said about it below under for mental representation.
Beyond Correlation. Representations are synthesized from parts by
chunking. A part is like a field in a record or a slot in
5. SUMMARY AND DISCUSSION a frame, for example, and the parts are chunked
Information with structure, such as language with its together by normalizing their sum (i.e., by superposi-
grammatical structure or concepts built of other con- tion; thresholded sum for the binary spatter code).
cepts by some logic, has traditionally been the The chunked representation—the holistic record—
domain of symbolic representation. Artificial neural correlates with its parts, as shown in Figure 4. When
nets as mathematical models of the brain need to han- the dimensionality is high, even a low correlation
dle such information. Out of this need has grown between a chunk and its parts is significant. This mat-
holistic representation. ters greatly in the analysis of chunked representa-
Holistic representation is fundamentally distrib- tions. Since the chunking operator (normalized sum)
uted whereas traditional representation, with its is commutative, the chunked record represents the set
dependence on records with fields, is local. Local of its parts.
representation is common also in the nervous system. Both single codewords and codewords combined
Primary sensory–motor functions and basic emotions by binding can serve as elements or parts for chunk-
are wired in genetically so that a nerve impulse in a ing. In the examples of this paper, some parts have

363
been formed by binding a field name and a field For example, we would expect the codewords for
value to each other (e.g., name⊗Pat), others by bind- similar names to be similar; Pat and Pam should cor-
ing analogous values to each other (e.g., Pat⊗Lee in relate quite highly.
Solution 2u), and also a random vector R has been The Scatter Code of Smith and Stanford (1990)
used as a part. The binary spatter code binds with bit- demonstrates this principle beautifully. Numbers are
wise XOR, the real HRR binds with circular convolu- encoded in random N-bit strings so that the Hamming
tion, and the complex HRR with coordinatewise distance between them grows stochastically with the
complex multiplication (adding together of phase difference between the numbers. The codewords for
angles). adjacent integers are only a few bits apart, but as we
Chunked codewords are analyzed by probing and move from one integer to the next, a few bits are cho-
clean-up. A codeword that has been used for binding sen at random and flipped. The expected correlation
(e.g., name) is used also for probing, and the probing between codewords decreases as a negative exponen-
operator is the inverse of the binding operator. The tial of the difference between the numbers, and the
probing operator for the binary spatter code is the codewords for very different integers are approxi-
same XOR as used in binding (cf. Decoding above), mately N/2 bits apart. In the multidimensional Scatter
for the real HRR it is inverse convolution (also called Code such codewords for the individual dimensions
‘correlation’ and realized by convolution with a are bound together with coordinatewise XOR (Stan-
“reflected” probe), and for the complex HRR it is ford & Smith, 1994).
complex division (realized by coordinatewise multi- As new codewords are built from existing ones,
plication with the probe’s complex conjugates). It is they, too, are random but not necessarily uncorre-
important mathematically that the probing operator lated. Codewords that share parts or that are made of
distributes over the chunking operator (with the real similar parts are correlated. For example, if we
HRR it distributes only approximately, but that is encode the second record of Figure 2 using an uncor-
good enough). related codeword female, the resulting PSmithF cor-
Probing of chunked codewords produces noisy relates with the first PSmith (δ = 0.25, ρ = 0.5; the
results, and the noise increases with the number of conventional records differ in only one bit out of 68,
chunked parts, K, according to Figure 4. However, and for them ρ = 0.97). So why don’t added levels of
the statistics of long, random codewords are such that composition eventually drive everything into some
the valid codeword closest to a noisy result is correct small part of the code space? Chunking has the ten-
with high probability. Therefore a system based on dency to do exactly that, but fortunately binding
holistic representation needs a clean-up memory that (with XOR) has the opposite tendency to scatter
realizes the nearest-neighbor method. The clean-up things about. It is significant mathematically that the
memory could be a table of valid codewords and a binding and chunking operators complement each
procedure for finding the best-matching table entry other in this way.
for any noisy codeword, or it could be an autoasso- Beyond Correlation. Correlation alone apparently
ciative neural net that stores valid codewords as point lacks the power to fully describe structure, such as
attractors. For the neural net to work well, however, found in language. In language research, at least, sta-
it must have large and regular basins of attraction, tistical approaches are only a part of the picture, and
because the amount of noise that needs cleaning up is logical and symbolic approaches are often the major
easily 35% (see Fig. 4). However, there is a limit to part. That is particularly true as regards grammar.
how much noise can be cleaned up reliably, which in Holistic representation allows mapping between
turn limits the number of parts that can be chunked at codewords that may meet this added need for struc-
once into a holistic record if such a record is to be tural description. A high correlation between code-
analyzable. words means that they are related—the codewords
Correlation Between Codewords. In the examples occupy some small part of the code space. But signif-
of this paper we have assumed a base set of random icant relations between structures—and codewords—
codewords name, Pat, Lee, sex, …, 66 that are can also exist when the codewords are uncorrelated;
uncorrelated. This idealization is useful in getting the when they are in totally different parts of the code
discussion started, but whichever way we start, we space. Then the relation is analogical. Holistic
soon deal with correlated codewords, because the records for (Pat, male, 66) and (Lee, female, 33)
idea in holistic representation is that similar meaning would be uncorrelated, whereas the structures clearly
should be reflected in the similarity of bit patterns. are related. The same three variables simply have

364
very different values. The relation between such References
records is by holistic mapping. Anderson, J.A. (1995) An Introduction to Neural Networks.
We already have two examples of holistic map- Cambridge, Mass.: MIT Press.
ping. The mapping T in Solution 2u is composed of Hinton, G.E. (1990) Mapping part–whole hierarchies into con-
nectionist networks. Artificial Intelligence 46(1–2):47–75.
analogous parts of PM6 and LF6 by binding and
Hinton, G.E., McClelland, J.L., and Rumelhart, D.E. (1986) Dis-
chunking—the same operations that build holistic tributed representation. In D.E. Rumelhart and J.L. McClel-
records. In Solution 3a, (33⊗PF3)⊗LF6 can be land (eds.) Parallel Distributed Processing, vol. 1; 77–109.
interpreted as a request to find in LF6 the part that Cambridge, Mass.: MIT Press.
Kanerva, P. (1994) The Spatter Code for encoding concepts at
corresponds to the 33 of PF3, in other words, the many levels. In M. Marinaro and P.G. Morasso (eds.), ICANN
analogous part. It is to be noted that these mappings ’94, Proceedings International Conference on Artificial Neu-
themselves are points of the common code space, so ral Networks, vol. 1; 226–229. London: Springer–Verlag.
Kanerva, P. (1995) A family of binary spatter codes. In F. Fogel-
that it is even possible to represent and study analo-
man-Soulie and P. Gallineri (eds.), ICANN ’95, Proceedings
gies between analogies within this system. International Conference on Artificial Neural Networks, vol.
How are holistic representations brainlike beyond 1; 517–522. Paris: EC2 & Cie.
how artificial neural nets at large are? They work Kanerva, P. (1996) Binary spatter-coding of ordered K-tuples. In
C. von der Malsburg, W. von Seelen, J.C. Vorbruggen, and B.
only with large patterns (N > 1,000) that are funda- Sendhoff (eds.), Artificial Neural Networks—ICANN 96 Pro-
mentally random, and information is distributed in ceedings; 869–873. Berlin: Springer.
the extreme, including the information about the Plate, T.A. (1994) Distributed Representations and Nested Com-
positional Structure. PhD thesis. Graduate Department of
structure of information (there are no fields). Systems
Computer Science, University of Toronto. (Available on
scale readily to brain-size, yet composition should be Internet Ftp-host: [Link] as Ftp-file: /pub/tap/
done in relatively small chunks (evidence for Miller’s [Link].Z)
7 ± 2). Brains are instruments of learning; encoding Pollack, J.P. (1990) Recursive distributed representations. Artifi-
cial Intelligence 46(1–2):77–105.
new structure with holistic representation is simple, Smith, D., and Stanford, P. (1990) A random walk in Hamming
and the mapping of codewords provides a potentially space. Proceedings 1990 International Joint Conference on
useful model of analogy. Neural Networks (IJCNN 90), vol. 2; 465–470.
Smolensky, P. (1990) Tensor product variable binding and the
In this paper we have described holistic representa-
representation of symbolic structures in connectionist sys-
tion within the framework of traditional representa- tems. Artificial Intelligence 46(1–2):159–216.
tion. However, producing the functions of traditional Stanford, P., and Smith, D. (1994) The multidimensional Scatter
representation is not the final goal although it is an Code: A data fusion technique with exponential capacity. In
M. Marinaro and P.G. Morasso (eds.), ICANN ’94, Proceed-
important criterion and a measure of credibility. The ings International Conference on Artificial Neural Networks,
ultimate challenge and goal is to capture and describe vol 2; 1432–1435. London: Springer–Verlag.
structure in a stream of data by binding, chunking, Touretzky, D.S. (1990) BoltzCONS: Dynamic symbolic struc-
tures in connectionist networks. Artificial Intelligence 46(1–
and holistic mapping—and by whatever other useful
2):5–46.
operations are found. An added desideratum is that it Touretzky, D.S., and Geva, S. (1987) A distributed connectionist
should be relatively easy for us to interpret the result- representation for concept structures. Proceedings of the
ing structure symbolically or logically if we so wish. Ninth Annual Conference of the Cognitive Science Society;
155–164. Hillsdale, NJ: Erlbaum.

Kanerva, P. (1997). Fully distributed representation. Proceedings of 1997

Real World Computing Symposium (RWC ’97, Tokyo, Jan. 1997), pp.
358–365. Tsukuba-city, Japan: Real World Computing Partnership.

365