An unscripted look at

scripting languages

Brian Kernighan
Department of Computer Science
Princeton University
A typical exploratory data analysis problem

data: thousands of lines like

8/27/1883 Krakatoa 8.8

5/18/1980 Mt St Helens 7.6
3/13/2009 Costa Rica 5.1

task: find all the events with magnitude greater than 6

how do you proceed?

1. do it by hand

2. write a program in [insert favorite programming

language here]
C version Awk version
#include <stdio.h> $3 > 6
#include <string.h>

int main(void) {
char line[1000], line2[1000];
char *p;
double mag;

while (fgets(line, sizeof(line), stdin) != NULL) {

strcpy(line2, line);
p = strtok(line, "\t");
p = strtok(NULL, "\t");
p = strtok(NULL, "\t");
sscanf(p, "%lf", &mag);
if (mag > 6) /* $3 > 6 */
printf("%s", line2);
}
return 0;
}
Over-simplified history of programming languages

• 1940's machine language

• 1950's assembly language
• 1960's high-level languages: Algol, Fortran, Cobol, Basic
• 1970's systems programming: C
• 1980's object-oriented: C++
• 1990's strongly-hyped: Java
• 2000's copycat languages: C#
• 2010's re-try: Scala, Go ??
Over-simplified history of programming languages

• 1940's machine language

• 1950's assembly language
• 1960's high-level languages: scripting languages:
Algol, Fortran, Cobol, Basic Snobol
• 1970's systems programming: C shell
• 1980's object-oriented: C++ Awk
• 1990's strongly-hyped: Java Perl, Python, PHP, Ruby, …
• 2000's copycat languages: C# Javascript
• 2010's re-try: Scala, Go ?? Dart ??
LISP, Scheme, functional languages

this page intentionally left blank

What's a scripting language?

"Scripting is a lot like obscenity.

I can't define it,
but I'll know it when I see it."
Larry Wall, creator of Perl
Scripting languages

• characteristics
– text as a basic data type
– regular expressions for text searching and manipulation
– associative arrays as a basic aggregate type
– minimal use of types, declarations, initialization, etc.
– usually interpreted instead of compiled

• examples
– shell
– Awk
– Perl, Python, PHP, Ruby, Tcl, Lua, …
– (VB|W|C|J)Script, PowerShell
– Javascript
Notation is important

"Language shapes the way we think and

determines what we can think about."
Benjamin Whorf, American linguist

Benjamin Whorf, 1897-1941

"A programming language that doesn't

change the way you think is not worth
learning."
Alan Perlis, Epigrams on Programming

Alan Perlis, 1922-1990

AWK – the first (or second?) scripting language

[Al Aho, Brian Kernighan, Peter Weinberger (Bell Labs, 1977)]

intended for simple data processing and analysis:

selection, validation:
"Print all lines longer than 80 characters"
length > 80
transforming, rearranging:
"Replace the 2nd field by its logarithm"
{ $2 = log($2); print }

report generation:
"Add up numbers in first field, print sum and average"
{ sum += $1 }
END { print sum, sum/NR }
Structure of an Awk program

• a program is a sequence of pattern-action statements

pattern { action }
pattern { action }
…
• a pattern is a regular expression, numeric expression, string
expression or combination
• an action is executable code, similar to C
• usage:
awk 'program' [ file1 file2 ... ]
awk -f progfile [ file1 file2 ... ]
• operation:
for each file
for each input line
for each pattern
if pattern matches input line
do the action
Awk features

• input is read automatically across multiple files

– lines are split into fields ($1, ..., $NF; $0 for whole line)
• variables contain string or numeric values (or both)
– no declarations: type determined by context and use
– initialized to 0 and empty string
– built-in variables for frequently-used values
• operators work on strings or numbers
– coerce type / value according to context
• associative arrays (arbitrary subscripts)
• regular expressions (like egrep)
• control flow statements similar to C: if-else, while, for, do
• built-in and user-defined functions
– arithmetic, string, regular expression, text edit, ...
• printf for formatted output
• getline for input from files or processes
Basic Awk programs, part 1

{ print NR, $0 } precede each line by line number

{ print $2, $1 } print field 2, then field 1
{ temp = $1; $1 = $2; $2 = temp; print } flip $1, $2
{ $2 = ""; print } zap field 2
{ print $NF } print last field

NF > 4 print if more than 4 fields

$NF > 4 print if last field greater than 4
/regexpr/ print matching lines (egrep)
$1 ~ /regexpr/ print lines where first field matches
Basic Awk programs, part 2

NF > 0 {print $1, $2} print two fields of non-empty lines

END { print NR } line count

{ nc += length($0) + 1; nw += NF } wc command
END { print NR, "lines", nw, "words", nc, "characters" }

length($0) > max { max = length($0); line = $0 }

END { print max, line } print longest line
Associative arrays (hash, dictionary, hashtable, map, ...)

• array subscripts can have any value, not just integers

input:
pizza 200
coke 50
beer 100
beer 150
pizza 500
beer 50
program:
{ amount[$1] += $2 }
END { for (name in amount)
print name, amount[name] }
output:
beer 300
pizza 700
coke 50
Some Awk lessons (1)

"One thing [the language designer] should

not do is to include untried ideas of his own."
C. A. R. Hoare, Hints on Programming Language Design, 1973

• mistakes are inevitable and hard to change

– function syntax
– concatenation syntax
– ambiguities, especially with >
– creeping featurism from user pressures
• standardization is hard
– there is a POSIX standard for Awk
– awk, gawk, mawk, tawk, busybox awk, ... all differ in places
• internationalization is hard
$1 ~ /[ - ]/ { ... }
Some Awk lessons (2)

• people use tools in unexpected, perverse ways

– compiler writing

– implementing languages

– object language
machine generated inputs stress a program differently than people do

– first programming language

Unexpected uses ...

Mon gourou --
I want to store all input lines of a big test file into one
variable. For example in this sample of my text file:

GATCTGATAAGCCCAGGCTTCAGAAGAGCTGTGAGACCTTGGCCAAGTCACT
TCCTCCTTCAGGAACATTGCAGTGGGCCTAAGTGCCTCCTGCGGGGACTGGT
AGTGGGGAGCGGTCATGCAATGAGTCCAATCCGGGTCAATAACAGTCAGTAG
ATCCAGCCCTATTAACGATCATCATTTCAGCTAGGTAAAAACACCTCGAGTC
AAGGAATGTGTCTGAATATTTTTCAGACATTAGTCCATTAAACTGCTTGCAA
I want to concatenate all the line into a unique variable, but my
file contains about 500,000 lines and when I use
var = var $0
the process take too much time to make the concatenation and
store the result in var.
Is there a specific algorithm in awk to store a big input in var?
Are scripting languages too slow?

"Premature optimization is the root of all evil"

Don Knuth
Volcano example in Perl and Python

while (<>) {
@w = split '\t';
print $_ if $w[2] > 6;
}

import sys
import string
line = [Link]()
while line != "":
wds = [Link]().split('\t')
if [Link](wds[2]) > 6.0:
print line,
line = [Link]()
How fast do they run? How big are they?
input: 1M lines, 21 MB

20

18

16

14

12

10

0
C Awk Perl Python C++ Java PHP Ruby Tcl Javascript
Text formatting example

• problem: format arbitrary text into lines of <= 60 characters

• by filling up successive lines as much as possible

• an example that combines text manipulation and a bit of

arithmetic
Awk text formatter
# format text into 60-character lines
/./ { for (i = 1; i <= NF; i++) addword($i) }
/^$/ { printline(); print "" }
END { printline() }

function addword(w) {
if (length(line) + length(w) > 60)
printline()
line = line space w
space = " "
}

function printline() {
if (length(line) > 0)
print line
line = space = ""
}
Javascript text formatter
var fs = require('fs');
var line = "";
var space = "";
var buf = [Link]([Link][2], 'utf-8');
words = [Link](/\n/g, ' ').trim().split(/ +/);

for (i = 0; i < [Link]; i++)

addword(words[i]);
printline();

function addword(w) {
if ([Link] + [Link] > 60)
printline();
line = line + space + w;
space = " ";
}
function printline() {
if ([Link] > 0)
[Link](line);
line = space = "";
}
How fast do they run? How big are they?
input: 155K lines, 22 MB
40

35

30

25

20

15

10

0
C C++ Perl Python Awk Java PHP Javascript Ruby Tcl
Word frequency count

• count the number of times that each distinct word appears

• Awk:

{ for (i = 1; i <= NF; i++)!

x[$i]++!
}!
END { for (i in x)!
print i, x[i]!
}!
Word frequency count: Python

import sys, string!

buf = [Link]()!
wordlist = [Link](buf) !
wd = {}!
for word in wordlist:!
if wd.has_key(word):!
wd[word] = wd[word] + 1!
else:!
wd[word] = 1!
for k, v in [Link]():!
print k, v!
Behavior of associative arrays

• note that it's necessary to test whether the name is in the table
• if not, have to insert it
• can't just increment

• this is different from Awk and Perl

How fast do they run? How big are they?
Opinions on scripting languages

• Perl
– in part a reaction to things missing from Awk
– "Perl is Awk with skin cancer" (Henry Spencer)
• Python
– "If you decide to design your own language, there are thousands of sort of
amateur language designer pitfalls." (Guido von Rossum)
• PHP
– in part a simplification of Perl
– "takes the worse-is-better approach to dazzling new depths" (Larry Wall)
• Ruby
– part Perl, part Smalltalk
– "it's just plain impossible to design a perfect language" (Yukihiro Matsumoto)
• Javascript
– in part a reaction to Java for applets?
– "Makes Javascript suck less" (marketing slogan for MochiKit)
Perl vs. Awk
• tradeoffs in Awk were made to keep it small and simple
• tradeoffs in Perl were made to make it powerful and expressive
• domain of applicability
– Awk is better for true 1-liners
– Perl scales to bigger programs, does system applications much better
• learning curve
– Awk is a lot simpler
• efficiency
– Perl is usually faster
• standardization
– there's only one Perl (?)
• program size, installation, environmental assumptions
– Perl is big, uses a big configuration script, takes advantage of the
environment
– Awk is small, uses no configuration script, does not try to adapt to the
environment
[Link]/224
Perl vs. Python

• most tradeoffs in Perl made to make it powerful and expressive

• most tradeoffs in Python made to make it small and interactive
• domain of applicability
– Perl does OS interactions well
– Python is simpler and cleaner
– Python's interactive mode is convenient
– Python is more extensible?
• efficiency
– about the same
• standardization
– there's only one Perl but it evolves
– there's only one Python but it evolves
• program size, installation, environmental assumptions
– both are big, use a big configuration script, take advantage of the
environment
– Python is somewhat smaller, but getting bigger all the time
[Link]/353
Why a language succeeds

• solves an important problem in a clearly better way

• culturally compatible and familiar

– C-like syntax helps
– easy to get started with

• environmentally compatible
– don’t have to buy into an entire new environment to use it
– e.g., can use standard Unix tools
– e.g., can link to existing C libraries

• portable to new environments

• open source, not proprietary

Why scripting languages succeed

• expressive: it's easy to write code

• efficient enough (usually)
• extensible (usually)
• portable, run everywhere

• good for data exploration, validation, glue, prototyping

• often good enough for production

• downsides:
– many errors only found at run time
e.g., mis-matched types, missing libraries, …
– creeping featurism: they're all getting bigger
– inconsistencies among similar languages (especially libraries)
– they do not scale to really big programs!
What makes Python successful?
• comparatively small, simple but rich language
– regular expressions, strings, tuples, assoc arrays
– clean (though limited) object-oriented mechanism
– reflection, etc.

• efficient enough
– seems to be getting better
• large set of libraries
– extensible by calling C or other languages
• embeddings of major libraries
– e.g., TkInter for GUIs
• open source with large and active user community
• standard: there is only one Python
– but watch out for Python 3, which is not backwards compatible

• a reaction to the complexity and irregularity of Perl?

One man's opinion

From r@[Link] Sun Jan 1 16:18:21 2006

Date: Sun, 1 Jan 2006 13:18:08 -0800
From: Rob 'Commander' Pike <r@[Link]>
To: Brian Kernighan <bwk@[Link]>

python is a very easy language. i think it's actually a good

choice for some things. awk is perfect for a line or two,
python for a page or two. both break down badly when used
on larger examples, although python users utterly refuse to
admit its weaknesses for large-scale programming, both in
syntax and efficiency.

-rob
The future of scripting languages

• they will continue to grow in importance

• they will continue to grow in number
though only a handful will be widely used
• everyone should know a few
• it doesn't matter which ones

• my current choices
Awk: the most bang for the buck (for small programs)
Python: broad utility, ease of use, readable, efficient
Perl, PHP, Ruby: all fine if you already know them
Javascript: client-side web programming
 “There will always be things we wish to
say in our programs that in all known
languages can only be said poorly.”

Alan Perlis