Commentary fPemrose: Emperor's new mind:

in undefended flank phenomena being logically, indeed nomically, independent.

That is, a being could have mathematical thoughts having
Jeremy Butterfield neither qualia nor indexical thoughts, and so on. (2) Since these
Philosophy Faculty, Cambridge University, Cambridge CB3 9DA, England threats are much debated in the philosophical literature, Pen-
rose has an undefendedflank:Might not the materialist philoso-
What a marvellous book! I discern three main ingredients. phers rebut his argument from the phenomenology of mathe-
1. The best kind of popular science: not just detailed and matical insight, in much the way they rebut the argument from
clear, but also forthcoming about unresolved issues. Setting qualia (e.g., Lewis 1990)?
aside the better-known issues in the foundations of quantum Two minor points. (1) Whatever consciousness is, it is a non
theory, examples include: the distinction between "good" and sequitur to infer (p. 408)fromits having evolved to its having a
"bad" uses of Cantor's diagonal argument (p. Ill); the recur- selective advantage, and so an active role. It might be a neces-
siveness of the Mandelbrot set (p. 125); self-energy in classical sary or nomic concomitant of something with such advantage,
electromagnetism (p. 189); determinism in general relativity (p. that imposes no or such little enough disadvantage as the weight
215); complexity theory and quantum computers (pp. 145, 402). of a polar bear's warm coat (cf. Jackson 1982). (2) It is a non
2. Various controversial arguments, mostly against strong AI sequitur to infer from the timelessness of mathematical truth to
("the mind is a digital computer"). The main argument here is there being no threat of causal paradox in the transmission of
based on the nonalgorithmic nature of mathematical insight, mathematical beliefs, backward in time (p. 446). Even if the
allegedly shown by Godel's theorem (especially pp. 108-12; truths are timeless, beliefs in them (and if distinct: their physical
417-18). correlates in brains) are in time. So such transmission threatens
3. An overarching speculation that two disparate problems - paradox, as backward causation usually does. [See Libet: "Un-
the reconciliation of quantum theory with relativity, and the conscious Cerebral Initiative and the Role of Conscious Will in
relation of mind to body - are relevant to one another. This is Voluntary Action" BBS 8(4) 1985.]
filled out in various ways. The most striking is by a happy
analogy with Penrose's work on tiling and quasicrystals: A
thought that surfaces in consciousness is both one of many
previously unresolved alternatives (cf. the reduction of the Computing the thinkable
state-vector, and quantum computers), and the solution to a
problem, involving global interactions of a characteristically David J. Chalmers
quantum kind, as the growth of a quasicrystal might be (pp. 434- Center for Research on Concepts and Cognition, Indiana University,
39; 446). Bloomington, IN 47405
Electronic mail: dave@[Link]
For such a tour deforce, criticism of specific points is bound to
seem niggling. But better that than panegyric. And better The main thesis of Penrose's book is that mental processes might
that than just scepticism about the speculations: That would be be nonalgorithmic. There appear to be three different argu-
no news to Penrose, who always expresses them cautiously. So I ments for this conclusion, which I will present in stripped-down
take up two major, and then two minor, points. form.
First, I am not convinced that Penrose's "Godel" argument 1. Flie argument from introspection. (1) Some mental pro-
against strong AI avoids the objections against his precursor, cesses are not algorithmic at a conscious level, therefore: (2)
John Lucas (1961). (Penrose cites some: I would urge adding Some mental processes are not algorithmic.
Lewis 1969; 1979.) Conscious of these objections, Penrose If this statement of the argument seems a little bald, it is
makes afinalattack (pp. 417-18). Transposing the argument to difficult to imagine what else might be meant by the numerous
Lucas's terms, it is: If Lucas's arithmetical output is that of a appeals to "intuition" and "judgment" (pp. 411-15; 418-23). It
Turing machine, then the machine table must be so complex is clear that a premise is missing here. Penrose wishes to exclude
that Lucas cannot survey it to check that it delivers only truths. from the start the possibility of conscious mental processes that
(For if he could, then he could "defeat" his own table by are algorithmic at a level too low to be apparent to conscious
constructing its Godel proposition.) But this is incompatible introspection. This is a dangerous assumption, as the recent
with the fact that in mathematics "we do not bow down to the proliferation of connectionist models demonstrates. These mod-
authority of some obscure rules that we can never hope to els have made familiar the notion that the level at which a system
understand. We must see that each step . . . can be reduced to is algorithmic might fall well below the level at which the system
something simple and obvious." (p. 418) Contraposing, Penrose carries semantic interpretation (Smolensky 1988). It is not a
denies that Lucas's arithmetical output is that of a Turing huge leap to image that in many systems, including the human
machine. I reply: The "but" is a non sequitur. Unsurveyable brain, the computational level might fall below the conscious
complexity of the machine table is, of course, compatible with level
mathematics' rigourous standards of proof. Connectionist models are not explicitly considered in the
Penrose's second argument against strong AI is based on the book under review, but on the face of it they would seem to fall
phenomenon of having "in aflash"a complex thought (pp. 418- into the class of "computational" models that Penrose would like
23); and his speculation that this is connected to state-vector to dismiss. It would be interesting to see Penrose declare an
reduction and quasicrystals. Penrose is mainly concerned with explicit position vis-a-vis these models. If he exempts them from
mathematical thoughts. Indeed, he eventually says that he takes his criticisms, then the force of his critique of algorithmic
the essence of consciousness to be the "seeing" of such a models is considerably weakened; if he wishes to dismiss these
necessary truth as logic and mathematics provide (p. 445). This models, too, his arguments will need to be considerably
[Link] "consciousness," though unusual, would be harmless strengthened.
were it not for the fact that Penrose briefly argues that other It must be conceded that the connectionist approach has not
phenomena more usually associated with "consciousness" are a yet had much success in modelling the kind of temporally
threat to strong AI, namely, qualia (pp. 14, 447), personal extended processing, such as mathematical thought, that Pen-
identity, and indexicality (pp. 27, 409, 448). The brief treatment rose considers. Nevertheless, other work within the "subsym-
of these threats engenders two problems. (1) You can get the bolic paradigm" has made some progress on these matters. In
impression that Penrose's argument involves a unitary notion of particular, Mitchell and Hofstadter (1990) have produced an
consciousness tying all these phenomena together. Not so: As interesting model of perception and analogical thought in an
far as I can see, Penrose says nothing against the "divide and abstract domain. In this model, high-level processes emerge
rule" idea that "consciousness" is an umbrella term, all these from the interaction of a number of small, low-level agents.

658 BEHAVIORAL AND BRAIN SCIENCES (1990) 13:4

. 3 7 8 :DDAC 53 4 97 9 5 7 , CD 3 3 /3D 3 1 7 CD 07A 3D C 4 75D D D:7 3 4 97 7 D7 C 8 C7 3 3 34 7 3D

:DDAC 53 4 97 95 7 D7 C :DDAC 9 0 2
 Commentary/Penrose: Emperor's new mind:

Under the influence of various pressures, the model is able to The idea of algorithmic processing lies at the core of modern
come up with "insights" that are similar in kind to those of a cognitive science for good reason. Anyone who succeeds in
mathematician. The high-level behavior of the model appears in overthrowing this idea will have effected a deep conceptual
no sense algorithmic, -yet it emerges from a completely com- revolution in the way we think about the human mind. Penrose
putational substrate. has given it his best, and has written a fascinating book along the
2. The argument from Gddel's theorem. (1) Humans can "see" way, but his arguments are a little thin for the weight they have
the truth of certain mathematical statements that lie outside the to bear.
bounds of any given formal system, therefore: (2) Human math-
ematical thought is not constrained by any given formal system.
This is an interesting variant on the argument of Lucas (1961).
Instead of focusing on the formal systems that specify a particu- Is mathematical insight algorithmic?
lar machine, Penrose (pp. 416-18) focuses on the formal systems
that might specify our mathematical thought. Because we have Martin Davis
the ability to "see" that the Godel sentence for a given system is Courant institute of Mathematical Sciences, New York University, New
York, NY 10012
true, the argument runs, we are using processes outside the Electronic mail: davism@csd11 .[Link]
system. On page 41.8, Penrose states: "When we convince
ourselves of the validity of Godel's theorem we not only 'see' it, Roger Penrose replies, "No," and bases much of his case on
but by so doing we reveal the very nonalgorithmic nature of the Godel's incompleteness theorem: It is insight that enables us to
'seeing' process itself." see that the Godel sentence, undecidable in a give formal system
This seems fallacious. We do not have to invoke any mystical is actually true; how could this insight possibly be the result of an
processes to explain this step; we do not even have to invoke algorithm? This seemingly persuasive argument is deeply
consciousness, as Penrose suggests. The reason we can "see" flawed. To see why will require looking at Godel's theorem at a
that Godel sentences are true is simply that we have a built-in somewhat more microscopic level than Penrose permits
faith that our mathematical systems are consistent. It would not himself.
be a difficult matter, in principle, to build such faith into an It will be helpful (though not essential to our argument) to
algorithmic machine. (And if Penrose would wish to argue that, place the discussion in terms of what is usually calledfirstorder
unlike machines, humans can repeat the "Godelization" process logic. This is just the formal system that embodies the elemen-
ad infinitum, ad transfinitum, the reply is that in practice the tary classical logic of and, or, not, implies, all, there exists. In a
Church-Kleene result on enumerating constructive ordinals precise formulation offirstorder logic, it is necessary to explain
puts as many limitations on humans as it does on machines. We when some particular formula F is to be taken to be a logical
are finite creatures, and we cannot continue to the ultimate consequence of a set of formulas ("premises") F. This can be
Omega.) done in two essentially different ways: semantically and syntac-
To gain his reductio of the notion of algorithmic thought, tically. In the semantic version, F is a logical consequence of F if
Penrose postulates a single algorithm for determining mathe- F is true no matter how the extra-logical symbols appearing in F
matical truth, shared by the mathematical community. Even to and F are interpreted, so long as all the formulas in F are true
one who believes that mind is algorithmic, this seems a little under that same interpretation. (Metaphorically: F is true in
strange. If we stay within the usual bounds of number theory, every Platonic world in which the formulas of F are true.) In the
analysis and the like, such an idea is perhaps plausible. As soon syntactic version, "rules of proof' involving the straightforward
as we move beyond these into more abstract strata of set theory manipulation of symbols are specified, and F is said to be a
and logic, disagreement about "truth" becomes rife. Some logical consequence of F if F can be obtained from F by some
mathematicians "see" that the axiom of choice is true; others finite number of applications of those rules (Penrose, p. 104
"see" that it is false. Moving further out, the continuum hypoth- gives some samples of such rules). In Godel's 1929 doctoral
esis and the axiom of constructibility are still more controversial. dissertation, he establishes his famous completeness theorem,
If such a "universal" algorithm exists, it is a fuzzy thing indeed; it which states that the semantic and the syntactic versions are
becomes less and less universal the further we travel from the equivalent. Moreover, this equivalence is largely independent
commonplace. This fuzziness alone is enough to defeat Pen- of the detailed manner in which rules of proof are specified.
rose's argument: A fuzzy algorithm cannot be Godelized! Godel's completeness theorem answered a question Hilbert
3. The argument from physical processes. (1) At the lowest had posed in his address at the Bologna mathematical congress
level, physical processes might not be algorithmically specifia- of 1928. Hilbert's Entscheidungsproblem for first order logic
ble. (2) Mental processes are dependent upon physical pro- was also raised in 1928 (in the famous textbook by Hilbert &
cesses, therefore: (3) Mental processes may be nonalgorithmic. Ackermann (1928), not at the Bologna conference as Penrose
This is an ambitious argument, but one which must hold if asserts), and called "the fundamental problem of mathematical
Penrose's other conclusions are to be sustained. It is nothing but logic." The problem was to give an algorithm for deciding
an attempt to subvert the force of Church's thesis about the whether a given formula was a logical consequence (in the
universality of algorithms. There are two clear weak spots. First, semantic sense) of a given (finite) set of premises. Hilbert
even if (1) holds, it would still be far from clear that such singled out first order logic for this attention presumably be-
microscopic nonalgorithmicity should make any difference on a cause it seemed clear that all mathematical reasoning could in
macroscopic level. It seems plausible to hold that even if principle be carried out in this formalism.l For the premises one
electrons don't behave algorithmically, neurons still might. takes an appropriate set of mathematical axioms; a mathematical
Penrose acknowledges this gap, but does little to bridge it. theorem is then simply a logical consequence infirstorder logic
Second and more serious, it seems to me that Penrose has in fact of those axioms. Since an argument based on the rules of proof of
provided very little evidence for (1). He gives an impressive first order logic can be checked in a completely algorithmic way,
demonstration of the nonclassical, nonintuitive nature of micro- we have no trouble understanding why mathematicians should
scopic physical phenomena, but he gives no clear justification of agree about proofs (p. 417) so long as they agree about the
why these things should have any bearing on their al- axioms (and so long as these axioms are finite in number or at
gorithmicity. For example, physical processes may well be least are specified by an algorithm).
nonlocal, but algorithms were never committed to locality in the In this context, Godel's incompleteness theorem (in a
first place. Algorithmic specifications have many degrees of strengthened form based on work of J. B. Rosser as well as the
freedom. Although the final verdict will be determined em- solution of Hilbert's tenth problem) may be stated as follows:
pirically, I doubt that Church's thesis will give in easily. There is an algorithm that, given any consistent set of axioms,

BEHAVIORAL AND BRAIN SCIENCES (1990) 13:4 659

. 3 7 8 :DDAC 53 4 97 9 5 7 , CD 3 3 /3D 3 1 7 CD 07A 3D C 4 75D D D:7 3 4 97 7 D7 C 8 C7 3 3 34 7 3D
:DDAC 53 4 97 95 7 D7 C :DDAC 9 0 2