TJC : A Tcl to Java Compiler

Mo DeJong
Mo DeJong Consulting
mo@[Link]
Abstract
A recent trend in corporate computing environments is the reimplementation of legacy systems on a Java based software
platform. Significant long-term savings can be realized through use of a common software platform and tools, but the
costs of software development and retraining of engineers remains high. If a legacy system can be ported to a Java
environment without having to rewrite existing code, then most software development and retraining costs can be
avoided. A large semiconductor manufacturer recently faced just such a challenge and decided to evaluate Jacl (Tcl
interpreter written in Java) as a migration tool for a large legacy system implemented in Tcl. Jacl was found to be
satisfactory in all areas except one, runtime execution speed. The native version of Tcl contains a runtime compiler and
execution engine, while Jacl supports only interpreted execution. As a result, native Tcl executes code from 10 to 50
times faster than Jacl. This paper introduces TJC, a Tcl to Java compiler that converts Tcl procs into Java bytecode and
closes this performance gap. In many cases, TJC compiled Tcl code executes more quickly than the same code running
in native Tcl. This paper describes TJC's initial design and implementation, demonstrates code generated for simple
examples, and describes optimizations added as the compiler has matured into production ready software. In addition,
performance of TJC compiled code is compared to native Tcl, and to other scripting languages implemented in Java.

Keywords: Tcl, Java, JVM, Scripting, Compiler, Bytecode, Optimization

1 Design Goals performance results.

The TJC compiler was designed to optimize for runtime Modern JVMs go to great lengths to execute Java
execution speed. Execution time required to compile Tcl bytecode efficiently, compiling to native machine code in
procs, as well as runtime memory usage, were secondary many cases. Sun's Hotspot compiler is able to inline
considerations. The targeted execution environment was a methods, predict branches, and dynamically recompile
server system running JDK 1.4, with sufficient memory code based on actual usage. Unfortunately, these
and CPU resources to handle most tasks. Since compile optimizations would be of little value to a Tcl bytecode
time was a secondary concern, the compiler could make interpreter loop implemented in Java. A significant
multiple passes and generate the fastest possible code for a percentage of an application's execution time could be
specific usage. Compatibility with native Tcl was a design spent just decoding Tcl bytecode instructions and then
requirement and has been maintained in all areas except jumping into and out of code that implements specific
one. TJC will generate inlined logic for Tcl primitives like instructions [3, 4, 5, 6, 7]. Option A was rejected on these
set, if, for, lindex, and others. The compiler assumes grounds.
that the user will not redefine these built-in Tcl primitives
at runtime. Option B was briefly considered as a solution to the Tcl
bytecode execution issues described above. The existing
Early on, three options were considered: compiler from native Tcl could be ported to Java, but it
would be modified to emit Java bytecode equivalents for
A) Duplicate native Tcl's bytecode compiler and each Tcl bytecode instruction. This approach would avoid
execution engine using CPU resources to decode Tcl bytecode instructions
at runtime. The Hotspot compiler could then convert entire
B) Duplicate native Tcl's bytecode compiler but emit
compiled command implementations to native code and
Java bytecode
aggressively inline utility methods. Java libraries that
C) Design new Tcl compiler and emit Java source could significantly simplify this type of implementation
code are freely available [8, 9]. However, option B was rejected
Option A was the most straightforward. Native Tcl because it would be too complex to implement, debug, and
contains a compiler that emits Tcl bytecode and a runtime modify.
execution engine that interprets Tcl bytecode instructions
[1, 2]. Although porting native Tcl's compiler and Option C involves emitting Java source code and
execution engine to Java would take time, this approach converting to Java bytecode using a Java compiler like
involved few risks or unknowns. The problem with option javac. This approach would require quite a bit of
A was that it was unclear if running a Tcl bytecode implementation effort, as new compiler and runtime
interpreter on top of the JVM would produce acceptable support modules would need to be implemented from
scratch. This approach would also generate a relatively Batch mode is invoked from the command line via a script
large number of Java bytecode instructions, as compared named tjc.
to the more compact Tcl bytecode instructions.
Nonetheless, option C was chosen because it had a $ tjc [Link]
number of important advantages.
The invocation above will scan the Tcl files indicated in
A new compiler could take advantage of optimizations [Link] and compile each statically defined proc.
that were not even considered in the native Tcl compiler. The compiler will generate a Java source file for each proc
Java bytecode would be executed directly in the JVM, so and then invoke javac to compile to Java bytecode.
there would be no runtime overhead associated with a Tcl Finally, Java's jar tool is invoked to create a pair of Jar
instruction decode and execute loop. Option C would be files. The [Link] file contains compiled Java class
easy to debug, since any Java source code debugger could files. The [Link] file contains generated Java
be used to step through generated code. Each Tcl proc source files. A user would add both Jar files to the
would be mapped to a Java class, each Tcl proc invocation CLASSPATH and then start the Jacl shell.
would be mapped to a Java method invocation, and each
Tcl variable frame would be mapped to a Java stack 2.2 Runtime Mode
frame. In addition, Java profiling tools could be used to
profile generated code in terms of time taken by each Tcl
command.
TJC Janino Jacl
Readability and transparency were important factors in
choosing option C. Java source code emitted by the
compiler would be human readable and regression
testable. The importance of being able to easily understand
Tcl Java Java
compiler output cannot be understated. Many tricky
problems were solved without the aide of a runtime or a proc source bytecode
debugger, simply by looking at the Tcl input and the Java
output. Optimizations were similarly easy to visualize
when working directly with emitted source code.
In runtime mode, the compiler runs in a separate thread.
2 Dual Implementations Unlike native Tcl, procs are not automatically compiled
when defined in Jacl. The user must request that a specific
The TJC compiler supports two compilation modes. Batch proc be compiled. First, the user loads the TJC package
mode is used to compile all the procs defined in a set of into Jacl, then the TJC::compile command is invoked to
Tcl files into a single Jar file. Runtime mode is used to compile a proc.
compile a specific Tcl proc into Java bytecode. Runtime
mode consumes memory and CPU resources, so startup % proc hello {} {return "Hello World"}
time could be affected. Batch mode does all compilation % package require TJC
% TJC::compile hello
off-line, so only minimal CPU and memory resources are
required at runtime. Batch mode is particularly useful for Once compilation is finished, the compiled
large libraries of Tcl code. Runtime mode is most useful implementation of hello replaces the original. Behind the
when a small number of procs are to be compiled, or when scenes, TJC makes use of the Janino embedded Java
the procs are not defined until runtime. compiler [10] to convert generated Java source into
bytecode. Generated Java bytecode is then loaded into the
2.1 Batch Mode current interpreter.

TJC is implemented entirely in Tcl code. Modules are

organized as follows:
TJC Javac Jar
(Shared Modules)
[Link]
[Link]
[Link]
Tcl Java Java Jar [Link]
proc source bytecode file
(Runtime Modules)
[Link]

(Batch Modules)
 [Link] throws TclException
[Link] {
const0 = [Link](
Since TJC is written in Tcl, one can use the compiler to "Hello World!");
compile itself and realize a significant performance [Link]();
improvement. TJC compiling itself in interpreted mode [Link]();
}
can take 25 minutes, but a compiled version of TJC can
} // end class HelloCmd
compile itself in 2.5 minutes.
In this generated code, the HelloCmd extends
3 Generated Code Examples [Link] and implements the Command
interface. Jacl will invoke the cmdProc() method when
This section presents some simple Tcl procs and the Java
the hello command is called in the interpreter. Each class
code generated by TJC. Since generated code can be
generated by TJC includes code to push a call frame and
verbose, only the first example will include the full Java
then pop it off the stack when the method is finished. The
class output.
proc hello invokes just one Tcl command, it is translated
into three Java statements in the emitted code. Note the
3.1 Hello World
statement [Link](const0), it sets the
Input: interpreter result to the constant string "Hello World!".
Finally, the code returns and the method completes
proc hello {} { normally.
return "Hello World!"
} 3.2 Command Invocation
Output:
This example shows how a Tcl command would be
// TJC implementation of hello invoked from a compiled proc. The output in this example
import [Link].*; includes just the generated code for the command
invocation. The generated class defines three constant
public class HelloCmd strings, one for each command argument.
extends [Link] {
public void cmdProc( Input:
Interp interp,
TclObject[] objv) proc foolen {} {
throws TclException string length "foo"
{ }
if (!initCmd) { initCmd(interp); }
CallFrame callFrame = Output:
[Link](
interp, [Link]); // Snippet of [Link]
try {
if ([Link] != 1) { { // Invoke: string length "foo"
throw new TclNumArgsException( TclObject[] objv0 =
interp, 1, objv, ""); [Link](interp, 3);
} try {
{ // Invoke: return "Hello World!" TclObject tmp1;
[Link](); // Arg 0 constant: string
[Link](const0); tmp1 = const0;
if ( true ) { return; } [Link]();
} // End Invoke: return objv0[0] = tmp1;
} catch (TclException te) { // Arg 1 constant: length
[Link](interp, tmp1 = const1;
te, "hello"); [Link]();
} finally { objv0[1] = tmp1;
[Link](interp, // Arg 2 constant: "foo"
callFrame); tmp1 = const2;
} [Link]();
} objv0[2] = tmp1;
[Link](interp, null, objv0, 0);
TclObject const0; } finally {
[Link](interp,
protected void initConstants( objv0, 3);
Interp interp) }
} // End Invoke: string // Binary operator: 0 == 1
ExprValue tmp0 = new ExprValue();
The code above invokes the string Tcl command. The [Link](0);
[Link]() method calls the cmdProc() method in ExprValue tmp1 = new ExprValue();
[Link](1);
the StringCmd class, passing an array of TclObject
[Link](interp,
arguments. Most of the code in this invocation is needed TJC.EXPR_OP_EQUAL, tmp0, tmp1);
to populate the argument array and maintain Tcl's // End Binary operator: ==
reference counting rules for TclObject arguments. boolean tmp2 =
( [Link]() != 0 );
3.3 Inlined String Command if ( tmp2 ) {
[Link]();
Most Tcl command invocations are implemented like the } else {
[Link]();
previous example, but TJC is able to emit inlined code for }
a number of built-in Tcl commands. This example shows } // End Invoke: if
an inlined call to Tcl's string command. Again, the
output in this example includes just the generated code for The ExprValue class manages the details of maintaining
the command invocation. The generated class defines just the expression result type and its value. The expression
one constant, the string "foo". result is converted to a Java boolean and one of the
branches of the if statement is taken. If this example had
Input:
included Tcl commands in the if block, instead of a
proc foolen {} { comment, then these commands would appear before the
string length "foo" first [Link]() call.
}
4 Optimizations
Output:
This section describes some of the most important
// Snippet of [Link]
optimizations implemented in TJC. These examples are
{ // Invoke: string length "foo" simplified and make use pseudo-code, see [11] for detailed
int tmp0 = [Link]().length(); examples that include complete Java source code.
[Link](tmp0);
} // End Invoke: string 4.1 Shared Constants
This inlined string command is significantly less Assume the following Tcl proc is defined.
complex when compared to invoking the string
command at runtime. The inlined code avoids allocating proc hello {} {
an array, populating the array, incrementing and return "Hello World!"
}
decrementing ref counts, and array cleanup and release.
The inlined code is efficient and is easily optimized by the If this proc was interpreted in Jacl, the return command
Hotspot compiler. would be parsed from a string:
3.4 Inlined If Command "return \"Hello World!\""

This example shows how a Tcl if command is converted Jacl would parse these word elements into an array and
to inlined Java code and how a simple expression is then lookup and invoke the return command. Pseudo-
evaluated. code for this command might look like:
Input: TclObject[] objv = new TclObject[2];
objv[0] = [Link](
proc iftrue {} { "return");
if {0 == 1} { objv[1] = [Link](
# no-op "Hello World!");
} [Link](interp, objv);
}
The first and most obvious optimization to apply here is to
Output: avoid allocating two new TclObject values each time the
command is invoked. TJC creates a pool of shared
// Snippet of [Link]
constants and then uses these constants each time a value
{ // Invoke: if {0 == 1} ... is accessed inside a compiled proc. Pseudo-code might
look like: proc setme {} {
set i 0
TclObject[] objv = new TclObject[2]; set j $i
objv[0] = const0; }
objv[1] = const1;
[Link](interp, objv); The code above might be mapped to operations like:

This example assumes that the constants have already setVar("i", const0);
been initialized, another method would be emitted to do setVar("j", getVar("i"));
that.
In the pseudo-code above, the local variable i is accessed
void initConstants() { twice and j is accessed once. In interpreted mode, Jacl
const0 = [Link]( stores local variables in a hashtable. While this approach
"return"); is flexible, it can quickly become a performance problem
const1 = [Link]( because of the sheer number of hashtable searches.
"Hello World!");
}
Compiled local variables avoid a hashtable search on each
access by saving variables in an array. Each local variable
4.2 Cached Command Lookup name is associated with an integer array index. Pseudo-
code to allocate such an array might look like:
The next optimization that could be applied to the hello
proc would be to cache a reference to the command, Var[] compiledLocals = new Var[2];
instead of looking it up by name for each invocation. TJC compiledLocals[0] = new Var("i");
implements this optimization by passing a cached compiledLocals[1] = new Var("j");
command reference when invoking a command. This
reference would be defined as an instance variable in the Then, logic for the setme proc might look like:
generated class, so that it would be saved from one
setVar(compiledLocals[0], const0);
invocation to the next. setVar(compiledLocals[1],
getVar(compiledLocals[0]));
class HelloCmd {
Command ccmd0 = null;
4.5 Omit Unused Results
...
} A Tcl proc can return either an empty result or a specific
value. If a proc does not return a value, then the result of
Then, method invocation code might look like: the last command in the proc is returned as the result.
Consider the following:
TclObject[] objv = new TclObject[2];
objv[0] = const0; proc setme {} {
objv[1] = const1; set i 0
if ( ccmd0 == null ) { set j 1
ccmd0 = [Link]("return"); set k 2
} }
[Link](interp, objv, ccmd0);
The result of executing this proc is 2. Pseudo-code might
4.3 Inlined Commands look like:
Tcl's catch, expr, for, foreach, if, switch, and TclObject tmp;
while commands are special cases since these commands tmp = setVar("i", const0);
can contain other commands. When one of these setResult(tmp);
tmp = setVar("j", const1);
commands is found, TJC will inline the command and any setResult(tmp);
contained commands. TJC also includes inline support for tmp = setVar("k", const2);
other built-in Tcl commands, these are append, break, setResult(tmp);
continue, global, incr, lappend, lindex, list,
llength, return, and set. The result of this command can never be 0 or 1, so the
compiler need only emit the third call to setResult().
4.4 Compiled Local Variables TJC implements specific logic to detect when the result of
a command is not used. The compiler can then omit
Optimizing local variable access in a compiled proc is pointless result set operations. With this optimization
critical to efficient execution. Consider the following enabled, emitted code for the example above might look
procedure. like:
setVar("i", const0); of allocating and garbage collecting 1000 temporary
setVar("j", const1); ExprValue objects, a single object is allocated and
TclObject tmp = setVar("k", const2);
reused. Allocating and garbage collecting lots of
setResult(tmp);
temporary objects is a serious performance problem, even
for a modern JVM. This allocation change alone resulted
4.6 Expr Operations
in a 5x performance improvement for some examples.
The expr command and expression arguments to the for,
TJC is able to use compile time type information to further
if, and while commands are particularly important
optimize compiled expressions. Consider the
because expression evaluation can take up a large exprSetResult() method, it sets the interp result based
percentage of the total execution time. So, optimizing
on the type of the passed in ExprValue. TJC knows that
expression evaluation can have a significant performance
the result of a unary not operator is always an integer type,
impact. Consider the following example:
so int type logic from exprSetResult() can be inlined.
expr {!$v}
{ // Invoke: expr {!$v}
[Link]( getVar("v") );
This expression consists of a unary not operator and a
[Link](interp,
variable operand. Pseudo-code to evaluate this expression TJC.EXPR_OP_UNARY_NOT, tmp);
might look like: setResult( [Link]() != 0 );
} // End Invoke: expr
{ // Invoke: expr {!$v}
ExprValue tmp = new ExprValue(); In addition, TJC is able to inline logic from
[Link]( getVar("v") );
exprUnaryOperator(), to optimize the common case
[Link](interp,
TJC.EXPR_OP_UNARY_NOT, tmp); where the TclObject operand contains an integer value.
[Link](interp, tmp);
} // End Invoke: expr { // Invoke: expr {!$v}
TclObject otmp = getVar("v");
Tcl's expression evaluation logic is tricky, the variable if ( [Link]() ) {
[Link](
operand could contain an int, double, or string value. TJC
[Link] == 0 );
handles each of these input types in the } else {
exprUnaryOperator() method. The logic above is less [Link]( otmp );
than optimal, since a new ExprValue object is allocated [Link](interp,
each time the expression is evaluated. TJC addresses this TJC.EXPR_OP_UNARY_NOT, tmp);
issue by allocating temporary ExprValue objects at the }
beginning of a compiled proc. Pseudo-code might look setResult( [Link]() != 0 );
} // End Invoke: expr
like:
(At the beginning of the method) Applying these expression optimizations results in a
significant improvement in runtime performance. The
ExprValue tmp = new ExprValue(); Hotspot compiler, particularly in the server configuration,
does a very good job of optimizing the code above. In
... some cases, the code above runs only slightly slower than
Java code using typed local variables.
{ // Invoke: expr {!$v}
[Link]( getVar("v") );
[Link](interp, 5 Regression Testing
TJC.EXPR_OP_UNARY_NOT, tmp);
[Link](interp, tmp); TJC has been successful in large part due to the
} // End Invoke: expr effectiveness of the regression test suite designed and
implemented along with the compiler. The regression test
This change might not seem like a big deal, but it can have suite is called tjcruntime, it is available via CVS. Every
a huge impact on performance. Consider the following Tcl language feature is extensively tested by the suite.
loop, with an expr command in the body. Native Tcl includes a regression test suite, but it could not
be used directly since TJC supports multiple compilation
for {set v 0} {$v < 1000} {incr v} { options that need to be tested individually. Instead, many
expr {!$v}
tests from the native Tcl test suite were incorporated into
}
tjcruntime.
Allocating an ExprValue at the beginning of the
command moves the allocation outside of the loop. Instead When run, the tjcruntime test suite executes about
17,000 tests. There are 1,900 individual tests, each is
compiled with 9 different option configurations. isum lsum caller

6 TJC vs. Native Tcl Jacl 7600 3750 26500

TJC/Jacl 125 96 400
In this section a few simple Tcl procs are presented.
Execution time for an interpreted version of each proc is Native Tcl 220 181 1250
compared to a TJC compiled version and to the execution
time in native Tcl. These examples are very simple and are
by no means a complete comparison. For a detailed, 7 Java Scripting Languages
feature by feature comparison of TJC and native Tcl, see
[12]. In this section, timing results from some other scripting
languages implemented in Java are compared to plain Java
6.1 Int Sum Loop and to TJC. The isum, lsum, and caller examples from
the previous section were coded in each language. Only
proc isum { num } { the timing results (in uSecs) are presented here, see [13]
set sum 0 for full source code.
for {set i 0} {$i < $num} {incr i} {
incr sum $i
} isum lsum caller
return $sum Java 6 12 25
}
TJC/Jacl 125 96 400
% isum 1000
499500 Jython 120 100 390
Pnuts 146 880 550
6.2 List Sum Loop
Groovy 2100 1250 2650
proc lsum { elems } {
set sum 0 Beanshell 4087 9100 11200
foreach elem $elems {
incr sum $elem
} Both Groovy and Beanshell lack the ability to compile to
return $sum bytecode at runtime. Groovy (JSR-06) includes a batch
} mode compiler, but it generated classes that did not pass
the bytecode verifier. An attempt to evaluate Jruby 0.9.0
set elems [list]
for {set i 0} {$i < 1000} {incr i} { was made, but it did not work with JDK 1.4. Both Jython
lappend elems $i and Pnuts contain a bytecode compiler and both turned in
} performance numbers comparable to TJC.

% lsum $elems 8 Future Directions

499500
Performance of TJC compiled code is good, but could be
6.3 Call Command improved significantly. Many additional optimizations
have been identified, but they require time and funding to
proc caller {} {
implement.
for {set i 0} {$i < 1000} {incr i} {
callme "abcdefghijklmnop"
} For example, all expr operators should be inlined, but
} only the unary operators are inlined in the current
implementation. Command invocations could be
proc callme { str } { accelerated by saving argument arrays on the stack.
return [string index $str 0] Invocations that set a variable to the result of an inlined
} command could be optimized by emitting logic to skip
setting the interp result. Many scripts would run more
6.4 Timing Results quickly if the regexp and regsub commands were
inlined, since regular expression patterns could be
Timing results are given in uSecs. JDK 1.4.2 (-server) and compiled ahead of time.
Tcl 8.4.12 (compiled with gcc -O2) were run under
WinXP. Timing results are gathered after the Hotspot It may also be possible to significantly reduce the size of
compiler has optimized the code. emitted Java bytecode by using code templates that cover
the most common Tcl command invocations. Additional [Link]
research is needed to determine if these and other 6/full_papers/lewis/[Link]
optimizations would make TJC compiled code execute [2] Tcl bytecode optimization : some experiences : Kenny,
more efficiently. Sofer, Hobbs :
[Link]
9 Conclusions 2002/Tcl2002papers/kenny-bytecode/[Link]
[3] Catenation and specialization for Tcl virtual machine
The TJC compiler project has been an unqualified success. performance : Vitale, Abdelrahman :
The compiler is functionally complete, it supports both [Link]
batch and runtime compilation, and emits Java source coll=&CFID=15151515&CFTOKEN=6184618
code that is easily understood. Comparing execution times [4] Alternative dispatch techniques for the Tcl VM interpreter :
for interpreted Jacl to TJC shows that compiling results in Vitale, Zaleski :
an order of magnitude improvement. TJC outperforms [Link]
native Tcl in many cases, sometimes by a wide margin. [Link]
Timing results show that TJC tied both Jython and Pnuts [5] The Behavior of Efficient Virtual Machine Interpreters on
as the fastest scripting languages implemented in Java. Modern Architectures : Ertl, Gregg :
[Link]
The TJC project is an effective example of how [6] The Structure and Performance of Efficient Interpreters :
corporations can benefit from open source software. In Ertl, Gregg : [Link]
this case, porting the infrastructure from a legacy system [7] Effective Inline-Threaded Interpretation of Java Bytecode
to the Java environment was significantly less expensive Using Preparation Sequences : Gagnon, Hendren :
than reimplementing thousands of scripts. TJC has been [Link]
thoroughly tested and contains no known defects. The 2/[Link]
compiler is currently being evaluated for deployment in [8] BCEL: The Byte Code Engineering Library :
productions systems. [Link]
[9] Serp : An open source framework for manipulating Java
10 Availability bytecode : [Link]
[10] Janino : An embedded Java compiler :
Jacl 1.4.0 includes the TJC compiler, runtime modules,
[Link]
and complete documentation. Jacl can be downloaded
from: [11] TJC generated code examples :
[Link]
docs/TJC/[Link]
[Link]
[12] Comparing Tcl to TJC/Jacl :
Both Jacl and TJC are licensed under BSD like licenses. [Link]
For details, see the [Link] and [Link] [Link]
files included in Jacl. [13] Scripting Examples :
[Link]
11 Thanks ip

Many thanks go to the Austin based Advanced Process

Control Framework team at Advanced Micro Devices, Inc.
The TJC project would not have been possible without
their vision, funding, and patience. Thanks also go to New
Iron Systems for making it possible for this project to
move forward.

Many thanks go to One Moon Scientific for funding TJC's

runtime compiler implementation. The runtime compiler
work was made possible by NIH grant R01GM073053-02.
Thanks also go to the original Tcl compiler team at Sun
Labs and later Scriptics. Many good ideas that first
appeared in the Tcl compiler were subsequently
implemented in TJC.

12 References
[1] An On-the-fly Bytecode Compiler for Tcl : Lewis :