Canonical Abstract Syntax Trees - ScienceDirect
Canonical Abstract Syntax Trees - ScienceDirect
[Link]/locate/entcs
Antoine Reilles
CNRS & LORIA
Campus Scientifique, BP 239,
54506 Vandœuvre-lès-Nancy Cedex France
Abstract
This paper presents GOM, a language for describing abstract syntax trees and generating a Java implemen-
tation for those trees. GOM includes features allowing the user to specify and modify the interface of the
data structure. These features provide in particular the capability to maintain the internal representation of
data in canonical form with respect to a rewrite system. This explicitly guarantees that the client program
only manipulates normal forms for this rewrite system, a feature which is only implicitly used in many
implementations.
1 Introduction
Rewriting and pattern-matching are of general use for describing computations and
deduction. Programming with rewrite rules and strategies has been proven most
useful for describing computational logics, transition systems or transformation en-
gines, and the notions of rewriting and pattern matching are central notions in many
systems, like expert systems (JRule), programming languages based on rewriting
(ELAN, Maude, OBJ) or functional programming (ML, Haskell).
In this context, we are developing the Tom system [10], which consists of a
language extension adding syntactic and associative pattern matching and strategic
rewriting capabilities to existing languages like Java, C and OCaml. This hybrid
approach is particularly well-suited when describing transformations of structured
entities like trees/terms and XML documents.
One of the main originalities of this system is to be data structure independent.
This means that a mapping has to be defined to connect algebraic data structures, on
which pattern matching is performed, to low-level data structures, that correspond
to the implementation. Thus, given an algebraic data structure definition, it is
needed to implement an efficient support for this definition in the language targeted
by the Tom system, as Java or C do not provide such data structures. Tools like
ApiGen [13] and Vas, which is a human readable language for ApiGen input where
used previously for generating such an implementation to use with Tom.
However, experience showed that providing an efficient term data structure im-
plementation is not enough. When implementing computational logics or transition
systems with rewriting and equational matching, it is convenient to consider terms
modulo a particular theory, as identity, associativity, commutativity, idempotency,
or more problem specific equations [9].
Then, it becomes crucial to provide the user of the data structure a way to
conveniently describe such rules, and to have the insurance that only chosen equiv-
alence class representatives will be manipulated by the program. This need shows
up in many situations. For instance when dealing with abstract syntax trees in a
compiler, and requiring constant folding or unboxing operators protecting particular
data structures.
GOM is a language for describing multi-sorted term algebras designed to solve
this problem. Like ApiGen, Vas or ASDL [15], its goal is to allow the user of an
imperative or object oriented language to describe concisely the algebra of terms he
wants to use in an application, and to provide an (efficient) implementation of this
algebra.
Moreover, it provides a mechanism to describe normalization functions for the
operators, and it ensures that all terms manipulated by the user of the data structure
are normal with respect to those rules. GOM includes the same basic functionality
as ApiGen and Vas, and ensures that the data structure implementation it provides
are maximally shared. Also, the generated data structure implementation supports
the visitor combinator [14] pattern, as the strategy language of Tom relies on this
pattern.
Even though GOM can be used in any Java environment, its features have been
designed to work in synergy with Tom. Thus, it is able to generate correct Tom
mappings for the data structure (i.e. being formal anchors [6]). GOM provides
a way to define computationally complex constructors for a data structure. It
also ensures those constructors are used, and that no raw term can be constructed.
Private types [8] in the OCaml language do provide a similar functionality by hiding
the type constructors in a private module, and exporting construction functions.
However, using private types or normal types is made explicit to the user, while it
is fully transparent in GOM. MOCA, developed by Frédéric Blanqui and Pierre Weis
is a tool that implements normalization functions for theories like associativity or
distributivity for OCaml types. It internally uses private types to implement those
normalization functions and ensure they are used, but could also provide such an
implementation for GOM.
The rest of the paper is organized as follows: in Section 2, to motivate the
introduction of GOM, we describe the Tom programming environment and its fa-
cilities. Section 3 presents the GOM language, its semantics and some simple use
cases. After presenting how GOM can cooperate with Tom in Section 4, we expose
in Section 5 the example of a prover for the calculus of structures [3] showing how
the combination of GOM and Tom can help producing a reliable and extendable
A. Reilles / Electronic Notes in Theoretical Computer Science 176 (2007) 165–179 167
Here the definition of plus is specified functionally, but the function plus can
be used as a Java function to perform addition. Nat is the algebraic sort Tom
manipulates, which is mapped to Java objects of type Term. The mapping between
the actual object Term and the algebraic view Nat has to be provided by the user.
The language provides support for matching modulo sophisticated theories. For
example, we can specify a matching modulo associativity and neutral element (also
known as list-matching) that is particularly useful to model the exploration of a
search space and to perform list or XML based transformations. To illustrate the
expressivity of list-matching we can define the search of a zero in a list as follows:
boolean hasZero(TermList l) {
%match(NatList l) {
conc(X1*,zero,X2*) -> { return true; }
}
return false;
}
zero can be found in the list and the function returns true, false otherwise, since
no zero can be found.
Although this mechanism is simple and powerful, it requires a lot of work to
implement an efficient data structure for a given algebraic signature, as well as to
provide a formal anchor for the abstract data structure. Thus we need a tool to
generate such an efficient implementation from a given signature. This is what tools
like ApiGen [13] do.
However, ApiGen itself only provides a tree implementation, but does not allow
to add behavior and properties to the tree data structure, like defining ordered lists,
neutral element or constant propagation in the context of a compiler manipulating
abstract syntax tree. Hence the idea to define a new language that would overcome
those problems.
From this description, GOM generates a Java class hierarchy where to each
sort corresponds an abstract class, and to each operator a class extending this sort
class. The generator also creates a factory class for each module (in this example,
called BooleanFactory), providing the user a single entry point for creating objects
corresponding to the algebraic terms.
Like ApiGen and Vas, GOM relies on the ATerm [11] library, which provides an
efficient implementation of unsorted terms for the C and Java languages, as a basis
for the generated classes. The generated data structure can then be characterized
by strong typing (as provided by the Composite pattern used for generation) and
maximal subterm sharing. Also, the generated class hierarchy does provide support
for the visitor combinator pattern [14], allowing the user to easily define arbitrary
tree traversals over GOM data structures using high level constructs (providing
congruence operators).
ADT
Java+Tom
Java Java
javac
bytecode
In the following example, we see how the use of GOM for the data structure
definition and Tom for expressing both the invariants in GOM and the rewriting
rules and strategy in the program leads to a robust and reliable implementation for
a prover in the structure calculus.
When building a prover for a particular logic, and in particular for the system BV in
the structure calculus, one needs to refine the strategy of applying the calculus rules.
This is particularly true with the calculus of structure, because of deep inference,
non confluence of the calculus and associative-commutative structures.
We describe here briefly the system BV, to show how GOM and Tom can help
to provide a robust and efficient implementation of such a system.
Atoms in BV are denoted by a, b, c, . . . Structures are denoted by R, S, T, . . . and
A. Reilles / Electronic Notes in Theoretical Computer Science 176 (2007) 165–179 173
generated by
S ::= ◦ | a | S; . . . ; S | [ S, . . . , S ] | ( S, . . . , S ) | S
>0 >0 >0
where ◦, the unit, is not an atom. S; . . . ; S is called a seq structure, [S, . . . , S] is
called a par structure, and (S, . . . , S) is called a copar structure, S is the negation
of the structure S. A structure R is called a proper par structure if R = [R1 , R2 ]
where R1 = ◦ and R2 = ◦. A structure context, denoted as in S{ }, is a structure
with a hole. We use this notation to express the deduction rules for system BV, and
will omit context braces when there is no ambiguity.
The rules for system BV are simple, provided some equivalence relations on
BV terms. The seq, par and copar structures are associative, par and copar being
commutative too. Also, ◦ is a neutral element for seq, par and copar structures, and
a seq, par or copar structure with only one substructure is equivalent to its content.
Then the deduction rules for system BV can be expressed as in Figure 3.
Because of the contexts in the rules, the corresponding rewriting rules can be
applied not only at the top of a structure, but also on each subterm of a structure,
for implementing deep inference. Deep inference then, combined with associativity,
commutativity and ◦ as a neutral element for seq, par and copar structures leads
to a huge amount of non-determinism in the calculus. A structure calculus prover
implementation following strictly this description will have to deal with this non-
determinism, and handle a huge search space, leading to inefficiency [4].
The approach when using GOM and Tom will be to identify canonical repre-
sentatives, or preferred representatives for equivalence classes, and implement the
normalization for structures leading to the selection of the canonical representative
by using GOM’s hooks. This process requires to define the data structure first, and
then define the normalization. This normalization will make sure all units ◦ in seq,
par and copar structures are removed, as ◦ is a neutral for those structures. We
will also make sure the manipulated structures are flattened, which corresponds to
selecting a canonical representative for the associativity of seq, par and copar, and
also that subterms of par and copar structures are ordered, taking a total order on
structures, to take commutativity into account.
When implementing the deduction rule, it will be necessary to take into account
the fact that the prover only manipulates canonical representatives. This leads to
simpler rules, and allow some new optimizations on the rules to be performed.
(R; T , [R, T ] and (R, T )). In our implementation, we considered these constructors
as unary operators which take a list of structures as argument. Using GOM, the
considered data structure can be described by the following signature:
module Struct
imports
public
sorts Struc StrucPar StrucCop StrucSeq
abstract syntax
o -> Struc
a -> Struc
b -> Struc
c -> Struc
d -> Struc
...other atom constants
neg(a:Struc) -> Struc
concPar( Struc* ) -> StrucPar
concCop( Struc* ) -> StrucCop
concSeq( Struc* ) -> StrucSeq
cop(copl:StrucCop) -> Struc
par(parl:StrucPar) -> Struc
seq(seql:StrucSeq) -> Struc
To represent structures, we define first some constant atoms. Among them, the o
constant will be used to represent the unit ◦. The neg operator builds the negation of
its argument. The grammar rule par(StrucPar) -> Struc defines a unary opera-
tor par of sort Struc which takes a StrucPar as unique argument. Similarly, the rule
concPar(Struc*) -> StrucPar defines the concPar operator of sort StrucPar.
The syntax Struc* indicates that concPar is a variadic-operator which takes an
indefinite number of Struc as arguments. Thus, by combining par and concPar
it becomes possible to represent the structure [a, [b, c]] by par(concPar(a,b,c)).
Note that this structure is flattened, but with this description, we could also use
nested par structures, as in par(concPar(a,par(concPar(b,c)))) to represent
this structure. (R, T ) and R; T are represented in a similar way, using cop, seq,
concCop, and concSeq.
}
}
Once this function is provided, we can define the hooks for the variadic operators
concSeq, concPar and concCop. The hook for concSeq is the simplest, since the
structures are only associative, with ◦ as neutral element. Then the corresponding
hook has to remove the units, and flatten nested seq.
concSeq( Struc* ) -> StrucSeq
concSeq:make_insert(e,l) {
%match(Struc e) {
o() -> { return l; }
seq(concSeq(L*)) -> { return ‘concSeq(L*,l*); }
}
return ‘make_concSeq(e,l);
}
This hook only checks the form of the element to add to the arguments of the variadic
operator, but does not use the shape of the previous arguments. The hooks for
concCop and concPar are similar, but they do examine also the previous arguments,
to perform sorted insertion of the new argument. This leads to a sorted list of
arguments for the operator, providing a canonical representative for commutative
structures.
concPar( Struc* ) -> StrucPar
concPar:make_insert(e,l) {
%match(Struc e) {
o() -> { return l; }
par(concPar(L*)) -> { return ‘concPar(L*,l*); }
}
%match(StrucPar l) {
concPar(head,tail*) -> {
if(!(compareStruc(e, head) < 0)) {
return ‘make_concPar(head,concPar(e,tail*));
}
}
}
return ‘make_concPar(e,l);
}
The associative matching facility of Tom is used to examine the arguments of the
variadic operator, and decide whether to call the builtin construction function, or
perform a recursive call to get a sorted insertion.
As the structure calculus verify the De Morgan rules for the negation, we could
write a hook for the neg construction function applying the De Morgan rules as
in Section 3.2 to ensure only atoms are negated. This will make implementing the
deduction rules even simpler, since there is then no need to propagate negations in
the rules.
A. Reilles / Electronic Notes in Theoretical Computer Science 176 (2007) 165–179 177
Once the data structure is defined, we can implement proof search in system BV in
a Tom program using the GOM defined data structure by applying rewriting rules
corresponding to the calculus rules to the input structure repeatedly, until reaching
the goal of the prover (usually, the unit ◦).
Those rules are expressed using Tom’s pattern matching over the GOM data
structure. They are kept simple because the equivalence relation over structures is
integrated in the data structure with invariants. In this example, [] and () structures
are associative and commutative, while the canonical representatives we use are
sorted and flattened variadic operators.
For instance, the rule s of Figure 3 can be expressed as the two rules [(R, T ), U ] →
([R, U ], T ) and [(R, T ), U ] → ([T, U ], R), using only associative matching instead of
associative commutative matching. Then, those rules are encoded by the following
match construct, which is placed into a strategy implementing rewriting in arbitrary
context (congruence) to get deep inference, the c collection being used to gather
multiple results:
%match(Struc t) {
par(concPar(X1*,cop(concCop(R*,T*)),X2*,U,X3*)) -> {
if(‘T*.isEmpty() || ‘R*.isEmpty() ) { }
else {
StrucPar context = ‘concPar(X1*,X2*,X3*);
if(canReact(‘R*,‘U)) {
StrucPar parR = cop2par(‘R*);
// transform a StrucCop into a StrucPar
Struc elt1 = ‘par(concPar(
cop(concCop(par(concPar(parR*,U)),T*)),context*));
[Link](elt1);
}
if(canReact(‘T*,‘U)) {
StrucPar parT = cop2par(‘T*);
Struc elt2 = ‘par(concPar(
cop(concCop(par(concPar(parT*,U)),R*)),context*));
[Link](elt2);
} } } }
We ensure that we do not execute the right-hand side of the rule if either R or T
are empty lists. The other tests implement restrictions on the application of the
rules reducing the non-determinism. This is done by using an auxiliary predicate
function canReact(a,b) which can be expressed using all the expressive power of
both Tom and Java in a factory hook. The interested reader is referred to [5] for
a detailed description of those restrictions.
Also, the search strategy can be carefully crafted using both Tom and Java
constructions, to achieve a very fine grained and evolutive strategy, where usual
algebraic languages only allow breadth-first or depth-first strategies, but do not let
178 A. Reilles / Electronic Notes in Theoretical Computer Science 176 (2007) 165–179
the programmer easily define a particular hybrid search strategy. While the Tom
approach of search strategies may lead to more complex implementations for simple
examples (as the search space has to be handled explicitly), it allows us to define
fine and efficient strategies for complex cases.
The implementation of a prover for system BV with GOM and Tom leads not
only to an efficient implementation, allowing to cleanly separate concerns about
strategy, rules and canonical representatives of terms, but also to an implementation
that can be proven correct, because most parts are expressed with the high level
constructs of GOM and Tom instead of pure Java. As the data structure invariants
in GOM and the deduction rules in Tom are defined algebraically, it is possible
to prove that the implemented system is correct and complete with respect to the
original system [5], while benefiting from the expressive power and flexibility of
Java to express non algebraic concerns (like building a web applet for the resulting
program, or sending the results in a network).
6 Conclusion
We have presented the GOM language, a language for describing algebraic signatures
and normalization systems for the terms in those signatures. This language is
kept low level by using Java and Tom to express the normalization rules, and by
using hooks for describing how to use the normalizers. This allows an efficient
implementation of the resulting data structure, preserving properties important to
the implementation level, such as maximal subterm sharing and a strongly typed
implementation.
We have shown how this new tool interacts with the Tom language. As Tom
provides pattern matching, rewrite rules and strategies in imperative languages like
C or Java, GOM provides algebraic data structures and canonical representatives to
Java. Even though GOM can be used simply within Java, most benefits are gained
when using it with Tom, allowing to integrate formal algebraic developments into
mainstream languages. This integration can allow to formally prove the imple-
mented algorithms with high level proofs using rewriting techniques, while getting
a Java implementation as result.
We have applied this approach to the example of system BV in the structure
calculus, and shown how the method can lead to an efficient implementation for a
complex problem (the implemented prover can tackle more problems than previous
rule based implementation [5]).
As the compilation process of Tom’s pattern matching is formally verified and
shown correct [6], proving the correctness of the generated data structure and nor-
malizers with respect to the GOM description would allow to expand the trust path
from the high level algorithm expressed with rewrite rules and strategies to the Java
code generated by the compilation of GOM and Tom. This allows to not only prove
the correctness of the implementation, but also to show that the formal parts of
the implementation preserve the properties of the high level rewrite system, such as
confluence or termination.
A. Reilles / Electronic Notes in Theoretical Computer Science 176 (2007) 165–179 179
Acknowledgement
I would like to thank Claude Kirchner, Pierre Étienne Moreau and all the Tom
developers for their help and comments. Special thanks are due to Pierre Weis and
Frederic Blanqui for fruitful discussions and their help in understanding the design
issues.
References
[1] Comon, H. and J.-P. Jouannaud, Les termes en logique et en programmation (2003), master lectures
at Univ. Paris Sud.
URL [Link]
[2] de Jong, H. and P. A. Olivier, Generation of abstract programming interfaces from syntax definitions,
Journal of Logic and Algebraic Programming 59 (2004), pp. 35–61.
[3] Guglielmi, A., A system of interaction and structure, Technical Report WV-02-10, TU Dresden (2002),
To app. in ACM Transactions on Computational Logic.
[4] Kahramanoğulları, O., Implementing system BV of the calculus of structures in maude, in: L. A.
i Alemany and P. Égré, editors, Proceedings of the ESSLLI-2004 Student Session, Université Henri
Poincaré, Nancy, France, 2004, pp. 117–127, 16th European Summer School in Logic, Language and
Information.
[5] Kahramanoğulları, O., P.-E. Moreau and A. Reilles, Implementing deep inference in TOM, in:
P. Bruscoli, F. Lamarche and C. Stewart, editors, Structures and Deduction (2005), pp. 158–172, iSSN
1430-211X.
[6] Kirchner, C., P.-E. Moreau and A. Reilles, Formal validation of pattern matching code, in: P. Barahone
and A. Felty, editors, Proceedings of the 7th ACM SIGPLAN international conference on Principles
and practice of declarative programming (2005), pp. 187–197.
[7] Kirchner, H. and P.-E. Moreau, Promoting rewriting to a programming language: A compiler
for non-deterministic rewrite programs in associative-commutative theories, Journal of Functional
Programming 11 (2001), pp. 207–251.
[8] Leroy, X., D. Doligez, J. Guarrigue, D. Rémy and J. Vouillon, The Objective Caml system (2004),
[Link] .
[9] Marché, C., Normalized rewriting: an alternative to rewriting modulo a set of equations, Journal of
Symbolic Computation 21 (1996), pp. 253–288.
[10] Moreau, P.-E., C. Ringeissen and M. Vittek, A Pattern Matching Compiler for Multiple Target
Languages, in: G. Hedin, editor, 12th Conference on Compiler Construction, Warsaw (Poland), LNCS
2622 (2003), pp. 61–76.
[11] van den Brand, M., H. de Jong, P. Klint and P. Olivier, Efficient annotated terms, Software, Practice
and Experience 30 (2000), pp. 259–291.
[12] van den Brand, M., J. Heering, P. Klint and P. Olivier, Compiling language definitions: The ASF+SDF
compiler, ACM Transactions on Programming Languages and Systems 24 (2002), pp. 334–368.
[13] van den Brand, M., P.-E. Moreau and J. Vinju, A generator of efficient strongly typed abstract syntax
trees in java, Technical report SEN-E0306, ISSN 1386-369X, CWI, Amsterdam (Holland) (2003).
[14] Visser, J., Visitor combination and traversal control, in: Proceedings of the 16th ACM SIGPLAN
conference on Object oriented programming, systems, languages, and applications (2001), pp. 270–282.
[15] Wang, D. C., A. W. Appel and J. L. Korn, The zephyr abstract syntax description language, in: USENIX
Workshop on Domain-Specific Languages, 1997.