% $Id: pretzelbook.tex,v 2.1 1997/07/09 14:39:29 gaertner Exp $
%----------------------------------------------------------------------

%
% The Pretzelbook -- the user's manual for Pretzel v.2.0
%
% [LaTeX]
% (history at end)
%
% NB: This file inputs small pretzel examples files that normally
% reside in the languages/examples subdirectory of the distribution.
% These files are:
%
%    simpas.ft
%    simpas.fg
%    small-example.simpas
%    example-frame.tex
%
%


\documentclass{book}
\usepackage{pretzeldoc}
\usepackage{a4}
%\usepackage[german]{babel}
\usepackage{makeidx}

\makeindex
%
% Indexing:
%   \index{first key!second key!third key}
%   \index{sorted by@{\it looks like}}
%
\begin{document}



\title{The PretzelBook \\ {\normalsize second edition}}
\author{Felix G\"artner}

\maketitle


\tableofcontents


% ######################################################################

\chapter{Introduction}

\begin{quote}

  ``I do think however that it is important that you are given some
  control over the basic prettyprinting style, as it can get very
  frustrating to see your code systematically reformatted in a style
  that you dislike.''

  Marc van Leeuwen \cite{vanLeeuwen96:pp:control}

\end{quote}

\index{Leeuwen@van Leeuwen, Marc}


\section{Do Prettyprinting the \pretzel{} Way}

\subsubsection*{What is \pretzel{}?}

Welcome to \pretzel{}. \pretzel{} is a prettyprinter generator, i.e.~a
system hat builds prettyprinters. The good thing about \pretzel{} is
that it builds prettyprinters with full user control. The new thing
about \pretzel{} is that it doesn't build standalone prettyprinting
programs, but rather builds modules that can be turned into standalone
programs easiliy, but can also be incorporated into your own software
systems.

\index{prettyprinter!generator}
\index{user control!full}
\index{modules}
\index{prettyprinting!modules}


\subsubsection*{What is this book about?}

This book is the ultimate source of information about
\pretzel{}. It explains nearly all (known and unknown) facets of the
\pretzel{} system and should be usefull to anybody using the system.

\index{ultimate source}
\index{pretzel@\pretzel{}!ultimate source}


\subsubsection*{How should I read this book?}

First of all, I expect that you are aquainted with the issue of
prettyprinting. Why would you use a system like \pretzel{} if you
weren't?! (However, if you want to freshen up your
understanding of the subject matter, you should have a glance at
chapter \ref{chap:on:prettyprinting} at some pleasing time, which gives an
overview over the area of prettyprinting and the history of the
concepts behind it.)

Essentially, you should start with the first chapter, that is the
tutorial of \pretzel{}. This might be the only part of this document
that most people will read. Later chapters deal with the more
intricate parts of using \pretzel{} (like chapter
\ref{chap:pretzel:hacking}, ``\pretzel{} hacking'') or with combining
\pretzel{} with the popular programming tool \noweb{} (chapter
\ref{chap:pretzel:meets:noweb}, ``\pretzel{} meets \noweb{}'').

The final chapter is the reference manual of \pretzel{}. Consult this
section only in emergencies.



\subsubsection*{Where can I get \pretzel{}?}

You can probably get \pretzel{} through the site or person through
which you got this document. \pretzel{} is free software, i.e.~you can
redistribute it and/or modify it under the terms of the GNU General
Public License as published by the Free Software Foundation (either
version 2 of the License, or (at your option) any later version).

\index{pretzel@\pretzel{}!obtaining}
\index{GNU General Public License}
\index{Free Software Foundation}

You can get the latest version of the distribution by looking at the
following WWW address, which points to the \pretzel{} homepage:
 %
\index{homepage!Pretzel}
\index{Pretzel!homepage}
 %
\begin{center}
  \verb"http://www.iti.informatik.th-darmstadt.de/~gaertner/pretzel/"
\end{center}

The latest distribution resides at:
 %
\begin{center}
  \verb"http://www.iti.informatik.th-darmstadt.de/~gaertner/pretzel/code/"
\end{center}
 %
The current release number of \pretzel{} is 2.0. Look out for the
latest information on the \pretzel{} homepage, if you want to be up to
date.

\index{Pretzel!currect release}
\index{2.0}
\index{date!up to}


\subsubsection*{How do I install \pretzel{}?}

The distribution contains a file called {\tt README} that contains
further instructions how to install \pretzel{}. To date, \pretzel{}
runs on \UNIX{}isch like machines and has been tested under HP-UX, AIX
RS6000 platforms and under Linux. It uses the GNU {\tt g++} compiler
and the common \UNIX{} tools {\tt flex} and {\tt Bison}.

\index{readme@{\tt README}}
\index{Pretzel!installing}
\index{installing Pretzel}
\index{UNIX}
\index{HP-UX}
\index{AIX}
\index{RS6000}
\index{GNU@GNU {\tt g++} Compiler}
\index{flex}
\index{Bison}


\section{History}

The \pretzel{} system started off as a minor project at the Institut
f\"ur Theoretische Informatik at the Technical University of
Darmstadt, Germany early 1993. It was revised during a sabbatical at
Trinity College, Dublin in late 1993 and version 1.1 was finished back
in Darmstadt in the summer of 1994. In early 1996 encouragement from
the net led me to take up the project again and prepare release 2.0,
which main feature was the ability to interface with \noweb{}. This
release was out by christmas 1996. Work continues mostly on building
new prettyprinting grammars for several languages.

\index{Pretzel!history}
\index{Darmstadt}
\index{ITI}
\index{Trinity College, Dublin}
\index{Dublin}
\index{noweb}
\index{christmas 1996}


\section{Acknowledgements}


\pretzel{} is by far no finished system and a lot still needs to be
done.  However I have to thank firstly Joachim Schrod for initiating
this project and his encouragement to hang on to it for so long. I
also have to thank Lee Wittenberg and Norman Ramsey for their help on
release 2.0 and to Holger Uhr for pointing me to a bug. Thanks also to
Roger Kehr and all those other students whom I have discussed this
topic with (whether they wanted or not).  I am very grateful to the
Institut f\"ur Theoretische Informatik here in Darmstadt and to
Professor Waldschmidt for providing the computing resources for this
project. Release 2.0 comes as a christmas present for me and the
people on the {\sc USENET} {\tt comp.programming.literate} newsgroup,
who --- with their discussions --- have helped to improve my
understanding of this subject and --- with their citations --- also
have helped to impove this book by far.

\index{Schrod, Joachim} \index{Waldschmidt, Helmut}
\index{Wittenberg, Lee} \index{USENET}
\index{Ramsey, Norman} \index{comp@{\tt comp.programming.literate}}
\index{2.0} \index{newsgroup}
\index{Kehr, Roger}
\index{Uhr, Holger}
\index{students}
\index{ITI}



\section{Changes to second Edition}
\label{sec:secondEdition}

The second edition of the PretzelBook features a dramatically enhanced
chapter on interfacing with \noweb{} and minor changes to the chapter
on \pretzel{} hacking.








% ######################################################################

\chapter{Using Pretzel}
\label{chap:using:pretzel}

\begin{quote}

  ``I'm using \CWEB{} to write chapters for books, and really
  appreciate the pretty-printing and automatic indexing and
  cross-referencing.''

  Tim Kientzle \cite{kientzle:use:of:prettyprinting}

\end{quote}

\index{Kientzle, Tim}
\index{CWEB}
\index{indexing, automatic}


\noindent This chapter explains how to use \pretzel{} in an everyday
setting. It covers the things you need to know if you want to build a
\pretzel{} prettyprinter from an existing set of \pretzel{} input files
or if you want to modify an existing input or write a totally new one
from scratch. It assumes that the \pretzel{} system has already been
installed on your system so that you can invoke the \t{pretzel-it}
command from your command prompt. You don't need any prior experience
with \pretzel{} or \LaTeX{} to start, maybe just a slight idea of what
kind of things regular expressions and context free grammars are, what
prettyprinting is about and how your source code looks best in a
prettyprinted way.

\index{everyday setting}
\index{setting, everyday}


\section{Getting Started}

This section will give a first introduction to using \pretzel{}. It
will show how to build a prettyprinter for a simple programming
language, which will be a subset of standard \PASCAL{}. After reading
this section and doing the excercises, you will be able to build
arbitrary prettyprinters from the ready-to-go input files included in
the \pretzel{} distibution.


\subsection{A first Example}

The input to \pretzel{} will always be {\em two} files. Let's start
with these files first and look at \pretzel{} in action. So you might
startup your computer, invoke your favourite text editor and create
two files called \t{simpas.ft} and \t{simpas.fg}. The suffixes \t{.ft}
and \t{.fg} are the extensions that \pretzel{} expects for its input
files. But before we learn about what they stand for, fill in the
files with the following information:\footnote{You can spare yourself
  from typing the data if you look into the subdirectory
  \t{languages/examples} of the \pretzel{} distribution.}

\index{Pretzel!input files}
\index{simpas@{\tt simpas.fg}}
\index{simpas@\t{simpas.ft}}
\index{Pretzel!file extensions}
\index{languages@\t{languages/examples}}

\verbatiminput{simpas.ft}

Here's the second file, which should be called \t{simpas.fg}:

\verbatiminput{simpas.fg}

The language that is covered by these first \pretzel{} specifications
is \PASCAL{} with simple expressions of the form $a\leq b$ or $a=b$
(you can see which operators are allowed from the \t{pascal.ft} file).
Also, the only statements allowed are assignments, the
\res{if}-\res{then}-construct and the compound statement
(i.e.~statements enclosed within \res{begin} and \res{end}). So the
following (not prettyprinted) code would be valid in these
terms:\footnote{Of course you can't do much with this ``language'',
  but remember that it's only an introductory example. Once you've
  understood the basics of this section it'll be easy to extend the
  specifications to cover all of \PASCAL{}.}

\verbatiminput{small-example.simpas}



\subsection{Running \pretzel{}}

Now we are ready to start: Having these files in one of your
directories, simply type:
 %
\begin{verbatim}
    pretzel-it simpas simpaspp
\end{verbatim}
 %
\index{pretzelit@\t{pretzel-it}}
 %
This will cause your machine to rattle and hum and if all goes well,
it will return saying ``done.'' and leave an executable called
\t{simpaspp}. This file is the prettyprinter for the specified
language!

\index{rattle and hum}
\index{hum, rattle and}
\index{simpaspp@\t{simpaspp}}
\index{executable prettyprinter}
\index{prettyprinter!executable}


By default, the prettyprinter reads text from the standard input and
writes formatted code to the standard output. So try to feed the small
example to the program and look what it returns:
 %
\index{standard input}
\index{standard output}
 %
\begin{verbatim}
simpaspp <small-example.simpas
\end{verbatim}
 %
This will result in a few lines that will have the name ``pretzel''
written everywhere and look something like this:
 %
\begin{verbatim}
if $a=1$ then \pretzelindent{}$b:=1$\pretzeloutdent{};
\pretzelforce{}if $a<=1$ then\pretzelindent{}
\pretzelforce{}begin $b:=2$ end\pretzeloutdent{}
\pretzelforce{}else\pretzelindent{}\pretzelforce{}$b:=a$\pretzeloutdent{}
\pretzelforce{}
\end{verbatim}
 %
\index{Pretzel!example output}
 %
In effect, this is \LaTeX{} code that you can use within your own
documents.

\index{latex@\LaTeX}


\subsection{Using \pretzel{} Output}

There's a small document prepared to demonstrate the use of \pretzel{}
output in your own documents. It's the file \t{example-frame.tex}.

\index{Pretzel!output}
\index{using Pretzel output}
\index{exampleframe@\t{example-frame.tex}}
\index{own documents}
\index{documents, own}

\verbatiminput{example-frame.tex}

If you save the output of the prettyprinter \t{simpaspp} in a file by
redirecting the standard output (let's call this file
\t{small-example.tex}) and run \LaTeX{} on the frame, the resulting
document will look something like this:

\index{smallexample@\t{small-example.tex}}
\index{standard output}

\begin{ppcode}
  \input{small-example}
\end{ppcode}

Now what do you think of this? Is it what you expected? It is obvious
that this is the code that has been input to the prettyprinter. But it
is not obvious, how the output corresponds to the information written
down in the two input files to \pretzel{}. This will be the subject of
the next subsection. But, alas, you have worked your way through this
first section and know now, how to use \pretzel{} in it's simplest way.

\index{expectations}


\section{Carrying On}

This section will show you how the basic prettyprinting method works
that \pretzel{} uses. After reading this section you will be able to
understand roughly what the contents of \pretzel{} input files mean
and how to modify existing input files to go with your wishes.


\subsection{The Two Input Files}

The prettyprinting method used by a \pretzel{} prettyprinter is quite
simple. It reads the input, which usually is the plain source code,
cuts this text into small pieces called {\em tokens}, and tries to
rearrange them in such a fashion that the user will call \ae{}sthetic.
Of course, the relative order of the input tokens will (usually) not
be changed, but the amount of whitespace inbetween might be enlarged
or reduced, line breaks may be inserted at special places etc.  All
this is specified by rules that the user writes. These rules describe,
how tokens are to be put together and how they in turn should be
formatted.

\index{Pretzel!prettyprinting method}
\index{tokens}
\index{Pretzel!input files}

The two input files of \pretzel{} reflect this separation of concerns.
One file tells the prettyprinter which small tokens to expect and the
second file states, how they should be put together again and how this
should look. The first file is called the {\em formatted token file},
because it contains token descriptions. The second file is called the
{\em formatted grammar file}, because that's what you call a set of
such rules: a grammar.

\index{formatted grammar file}
\index{formatted token file}


\subsection{Formatted Tokens}

In the simple example from above the first file (\t{simpas.ft}) was
the formatted token file. Looking at it you'll see that all the
reserved words from our \PASCAL{} subset appeared there together with
all the bits that make up expressions (like the binary operators $<$,
$\leq$, and so on). Accompanying these strings were special words in
uppercase (like \t{IF}, \t{ELSE}) which are the symbolic names of
these tokens. A vertical line in place of such a name means, that the
name of this token should be the same as the one immediately following
the current one. From now on, these tokens are referenced only by
their symbolic name.

\index{symbolic names}
\index{tokens!symbolic names}

Most of these token definitions are quite simple, except those from
the last three lines. If you already know, how a token looks like in
the source code, you can just write it down and assign a symbolic name
to it. However, if you don't know exactly what it looks like (which is
the case with normal identifiers or strings for example), you'll have
to specify an input pattern that the prettyprinter must match in order
to find and name the token. This pattern is specified by using {\em
  regular expressions}.

\index{regular expressions}
\index{match iput patterns}
\index{handling identifiers}


\subsection{Regular Expressions}

A case in which you'll need this feature of \pretzel{} crops up with
any language using identifiers for example.
In \PASCAL{}, identifiers begin with a letter between `a' and `z' (in
upper- or lowercase) followed by zero, one or more letters or digits.
This is expressed by the regular expression
 %
\begin{quote}
\t{[a-zA-Z][a-zA-Z0-9]*}
\end{quote}
 %
Regular expressions are read from left to right. The squared brackets
mean: ``a character from this set or range''. A star attached to such
a range means: ``zero, one or more of these characters.'' The regular
expression for integer numbers looks like this:
 %
\begin{quote}
\t{[0-9]+}
\end{quote}
 %
The plus sign means ``one or more of these characters'', which means
one or more consecutive digits in this context.

This might be enough for a first introduction into regular
expressions. See the manual pages {\tt flex(1)} or {\tt flexdoc(1)}
for a complete reference on how to write regular expressions that
match more complex patterns.  However, it might be important to note
here that the order of the regular expressions in the input file is
important. When the prettyprinter reads its input it tries to match
the regular expressions starting from the top. If it finds one or more
patterns that match the input file, it will take the longest match
(counting the characters matched). If there are several matches with
equal length it will take the one that comes first in the file. Most
notably, the pattern for identifiers is found at the very end of the
file. If it had stood right at the beginning, all reserved words like
\t{if} and \t{else} would habe been matched as \t{ID} tokens and not
as \t{IF} or \t{ELSE}.

\index{flex} \index{flexdoc} \index{placing regular expressions}
\index{regular expressions!placement in formatted token file}
\index{formatted token file!placement of regular expressions}
\index{Bison}


\subsection{Formatted Grammar}

The above explanations should be enough about the formatted token
file. But formatted tokens are only half the input to \pretzel{}. The
other half is the formatted grammar file. It specifies the rules by
which these tokens are put together again.

\index{formatted grammar file}
\index{combining rules}
\index{prettyprinting grammar}

The rules are specified in form of a {\em context free grammar}. A rule
specifies what tokens to look for, to put together and what the
resulting `thing' is. Actually, a grammar specifies tokens too, only in
a more complex way: it tells how complex tokens are put together by
adding zero, one or more simpler tokens. These simple tokens may be the
ones from the formatted token file, but also new ones specified in the
grammar.\footnote{Token names can be chosen as you like. By convention,
use lowercase for normal tokens and all uppercase for tokens that appear
in the formatted token file.  Do not try to call a token \t{error}. See
section \ref{sec:debugging} to see why.} When such a rule is actually
executed, we say that the simpler tokens are {\em reduced} to a (more)
complex token.

\index{context free grammar} \index{grammar, context free}
\index{reducing tokens} \index{complex tokens} \index{tokens}

A rule contains a `resulting' token, a colon followed by a series of
tokens from which the one preceding the colon is made. A rule is
terminated by a semicolon. Several rules reducing to the same token
can be appended by using a vertical line. The example above showed
that an expression (token name is \t{exp}) can be built from the
simple tokens \t{ID} (an identifier) or a \t{NUM} token (a number), or
in turn can be made from two expressions that have a \t{BINOP} token
inbetween. This is really recursive! Similar things count for
statements (token name \t{stmt}). There is no restriction on the
order of the rules as they appear in the formatted grammar file, except
that the prettyprinter will always want to end up with the token
that is mentioned first in the file. In our example, the prettyprinter
will always end up with a \t{stmt\_list} token. If this isn't possible,
it will be unhappy and fail.

\index{colon} \index{semicolon} \index{vertical line} \index{recursion}

There are only two more things to say about formatted grammars:
firstly there are these strange looking constructs amidst curling
braces behind some of the rules, and secondly, there are these funny
\t{\%token} things at the top.

\index{token@\t{\%token} declarations}
\index{formatted grammar file}

The \t{\%token} things on the top of the formatted grammar file are
the {\em token declarations}.  Every simple token that is named within
the formatted token file has to be declared here. This is just a way
of telling the prettyprinter which tokens actually come directly from
the input. They have to be there in order for the prettyprinter to
work.

The other things inside the curling brackets are a little more
interesting: they define how the major aspects of the output will look
like. Major aspects are indentation and line breaks. This gets more to
the heart of the matter of prettyprinting and we'll see that the stuff
inside the brackets is as important as the rules outside. But this is
so important that it deserves a subsection of its own.

\index{attribute definitions}


\subsection{Prettyprinting with Format Instructions}

This subsection introduces the way in which the user controls
prettyprinter actions.

A lot of early prettyprinters were very simple: they read the
programming language source code and scanned it for special words
(like \res{begin}). If they found one of these patterns, they executed
a special action. As most of the common output devices at that time
were dot matrix line printers or dumb {\sc ASCII} terminals the
actions performed when a reserved word like \res{begin} were
encountered were simple too. If for example the writer of the
prettyprinter wanted a line break before the \res{begin} and a little
indentation too, he would code this directly into his program. He
would keep a local count of the amount of indentation, increment and
decrement it when appropriate and indent to this amount when breaking
lines.

\index{formatting instructions} \index{prettyprinting!history}

\pretzel{} works in a similar but slightly more flexible way. We have
seen that the formatted token file contains patterns that the
prettyprinter tries to match to the input. This is exactly what the
simple prettyprinters did, but nothing more. To implement a simple
prettyprinter, we would only need to specify the patterns for the
tokens and add some information, what the output device should do if
this pattern can be matched. An example will show, that this is not
enough, if you strive for full user control over prettyprinting.

For example look at the following \PASCAL{} code fragment.

\begin{outp}
\res{if} $i>1$ \res{then} $i\mathrel{\tt :=}i-1$;\pretzelforce
\res{if} $b\leq i$ \res{then} \pretzelindent \pretzelforce
$b\mathrel{\tt :=}b+1$ \pretzeloutdent\pretzelforce\res{else}
\pretzelindent $b\mathrel{\tt :=}b-1$; \pretzeloutdent
\pretzelforce
\end{outp}

Say the user wants the first statement (an \res{else}-less \res{if})
to be prettyprinted without a forced line break behind the \res{then}
keyword and the second statement (a simple
\res{if}-\res{then}-\res{else}) to have a line break behind the
keywords and additional indentation to the statements to enhance
readability. This is hardly expressible in terms of ``formatted
tokens'', because it would be necessary to go back in the output and
insert a line break within material that might have already been
output to the screen or to paper.


To specify this distinction you have to tell the prettyprinter
something about what \res{if}-\res{then}s are in difference to
\res{if}-\res{then}-\res{else}s. That's where the formatted grammar
comes into action.

If you look at the \t{simpas.fg} example above, you'll notice two
rules that handle these two different cases. Here are both rules again
to look at:

% should be changed if simpas.fg is altered

\begin{verbatim}
stmt    : IF exp THEN stmt          { $1 " $" $2 "$ " $3 " " indent $4
                                      outdent }
        | IF exp THEN stmt ELSE stmt
                                    { $1 " $" $2 "$ " $3 indent force
                                      $4 outdent force
                                      $5 indent force $6 outdent force }
;
\end{verbatim}

The second line is more interesting at the moment: it means that any
\t{IF} token followed by \t{exp}, \t{THEN}, \t{stmt}, \t{ELSE} and
\t{stmt} tokens will be reduced again to a \t{stmt} token. The attached
description in curled braces tells the prettyprinter to insert a forced
line break (\fmt{force}) behind the \t{THEN} token if this rules fires.
Additionally, the statement following the \t{THEN} should have a little
more indentation before it (\fmt{indent}) which should be taken back
afterwards (\fmt{outdent}). The same is specified for the second
statement following the \t{ELSE} token. What is done here is to specify
something like an attachment (or special attribute) of the token which
this rule reduces to (i.e.~the \t{stmt} token on the left hand side).
This attachment can use the attachments of the tokens on the right hand
side by referring to the sequence number in which they appear there.
This means that \t{\$1} references the attachment of the \t{IF} token,
\t{\$2} the attachment of the \t{exp} token and so on. All the
attachments between the curled braces are put together and form the
attachment of the newly formed token (the \t{stmt} token). Apart from
referring to the attributes of existing tokens, you can additionally
insert special formatting instructions (like \fmt{indent}, \fmt{outdent}
or \fmt{force}) and strings (like the \verb+" $"+) into the sequence
as ``extra'' attachments. All these attachments together form the attribute
of the new \t{stmt} token.

\index{indent} \index{outdent} \index{force} \index{attachments} \index{attributes} \index{grammar rules}

The special commands that have appeared above, e.g.~the \fmt{indent} or
\fmt{force} commands, are called {\em format commands} or {\em
formatting instructions}. Subsequently the prettyprinting method used by
\pretzel{} prettyprinters is called {\em prettyprinting with format
commands}. The main assumption is that we have a text formatter at hand
that uses {\em control sequences} (or {\em tags}) inside the actual text
body for specifying all different ways of formatting (such as fonts,
indentation, spacing, alignment). Another term for this way of
formatting is {\em in-text procedural markup} and systems using this
technique are commonly called {\em document compilers}\footnote{Common
examples of such typesetting systems are \TeX{} and the \UNIX{} tools
Nroff/Troff.}. The idea behind the prettyprinting method presented here
is to systematically insert special formatting instructions into the
source code which are left to a typesetter to interpret. The history
behind this method and the details of the algorithm are explained in the
special chapter ``On Prettyprinting'' (chapter
\ref{chap:on:prettyprinting}). From now on, simply think of formatting
instructions as actions that will be performed on the typeset output
before it appears on the screen or on paper.

\index{prettyprinting!with format commands} \index{text formatter}
\index{control sequences} \index{tags} \index{markup} \index{formatting}
\index{tex@\TeX} \index{UNIX} \index{Nroff} \index{Troff} \index{prettyprinting!idea}


\subsection{Formatting Instructions}

Now, we'll have a look at the formatting instructions that \pretzel{}
offers.

\index{formatting instructions}


\subsubsection*{Controlling Indentation}

\index{controlling indentation}
\index{indentation}

We have met three formatting instructions already: \fmt{indent},
\fmt{outdent} and \fmt{force}. They in effect control the amount of
indentation. The \fmt{indent} primitive increases the level of
indentation by one unit, \fmt{outdent} does the same in a decreasing
fashion. However, the new level of indentation only becomes visible
after a forced line break, which corresponds to the \fmt{force} command.

\index{force} \index{indent} \index{outdent}

Let's look at an example. Here we have a simple \res{while} loop
in \PASCAL{}. If the input sequence is
 %
\begin{inp}
while (i<size) do \fmt{indent} begin i:=i+1; end; \fmt{outdent}
\end{inp}
 %
the output will simply look like this:
 %
\begin{outp}
\res{while} $(\id{i} < \id{size})$ \res{do} \res{begin}
$\id{i} \ass{} \id{i} + 1; $ \res{end};
\end{outp}
 %
However, if a \res{force} command is inserted behind the \fmt{indent}
command, we'll get the expected output:
 %
\begin{outp}
\res{while} $(\id{i} < \id{size})$ \res{do} \myindent\myforce
\res{begin} $\id{i} \ass{} \id{i} + 1; $ \res{end}; \myoutdent
\end{outp}
 %
So the \fmt{indent} and \fmt{outdent} primitives cause only future
lines to be in- or ``out''den\-ted and have no effect on the present
line that is just processed. The indentation shows only if there
is a forced line break after the amount of indentation has been
changed.

Why do \fmt{indent} and \fmt{outdent} to behave in
this way? To understand this, we should consider the above example
without the keywords \res{begin} and \res{end} surrounding the
assignment. The result
 %
\begin{outp}
\res{while} $(\id{i} < \id{size})$ \res{do} \myindent\myforce
$\id{i} \ass{} \id{i} + 1; $ \myoutdent
\end{outp}
 %
really does look a bit stupid. In cases like this we would like
the controlled statement to stand on the same line as the header
because the program structure doesn't need to be visualized in
this simple case by indentation. The only thing one has to to here
is to remove the \fmt{force} command from the input sequence and
we will get:
 %
\begin{outp}
\res{while} $(\id{i} < \id{size})$ \res{do}
$\id{i} \ass{} \id{i} + 1; $
\end{outp}
 %
The interesting fact is that this doesn't change our global rule
that the body of a \res{while}-statement should be \fmt{indent}ed
from it's keyword. So the \fmt{indent} and \fmt{outdent} commands
can always be inserted before and after the controlled statement.
The only thing to worry about is whether or not to insert a
\fmt{force} (which is always recommended before a \res{begin},
since this nearly always starts a sequence of two or more
statements).

\subsubsection*{Controlling Line-breaks}

\index{line breaks}
\index{controlling line breaks}

Let's see, how you can control line breaking. Consider for example a
long block of statements in \PASCAL\ that contain extremely long names
for identifiers as in the following example:
 %
\begin{inp}
begin string_buffer[string_buffer_pointer]:=c;
string_buffer_pointer:=string_buffer_pointer + 1; end;
\end{inp}
 %
We don't want line breaks to appear within identifiers and we don't want
things like this to crop up when prettyprinting:
 %
\begin{outp}
\res{begin}
$\id{string\_buffer}[\id{string\_buffer\_pointer}]\ass\id{c};
\id{string\_buffer\_pointer}$\myforce
$\ass\id{string\_buffer\_pointer} + 1;$
\res{end};
\end{outp}
 %
Here, a statement is broken across two lines, but we would prefer line
breaks to occur only between two adjacent statements (and usually also
before the \res{end}) like in:
 %
\begin{outp}
\res{begin}
$\id{string\_buffer}[\id{string\_buffer\_pointer}]\ass\id{c};$
\myforce
$\id{string\_buffer\_pointer}\ass\id{string\_buffer\_pointer} +
1;$\myforce
\res{end};
\end{outp}
 %
So the conclusion is, that we have to tell the formatter, where it is
possible to break a line, if the current block of text should exceed the
right margin. This is done with the \fmt{break\_space} primitive. By
inserting a \fmt{break\_space} amidst two statements (and even after the
\res{begin} and before the \res{end}) the above problem cannot happen
any more. A \fmt{break\_space} tells the typesetter, to break a line at
this point, if he is looking for a place to break somewhere near it.

\index{breakspace} \index{format commands!breakspace}

But even now, a single statement might be too long to fit onto one line.
Consider for example a \res{while}-loop or an
\res{if}-statement with a very long expression that governs it,
as in:
 %
\begin{inp}
while (str\_buf[str\_buf\_ptr]<>c) and
(str\_buf\_ptr<=buf\_size) do
\end{inp}
 %
Inserting \fmt{break\_space}s before and after the ``\t{and}''
would seem to be a good solution to this line-breaking problem.
But breaking the line after the ``\t{and}'', say, would result in
 %
\begin{outp}
\res{while} $(\id{str\_buf}[\id{str\_buf\_ptr}]<>\id{c})$ \andsign
\myforce
$(\id{str\_buf\_ptr}\leq\id{buf\_size})$ \res{do}
\end{outp}
 %
what of course is intolerable. We want a long continuous line to be
indented a little if it is broken, but the amount of indentation for the
continuing line must be more as the ``normal'' indentation of controlled
statements inf conditionals for example. This  distinction visually
separates adjacent (and indented) statements from a single statement
that fills two lines. To this effect, there exists a new format command,
namely the {\em optional line break} command \fmt{opt}. There are ten
different \fmt{opt}s that specify the badness of a line break at the
point of it's occurence. They are called \fmt{opt0}, \fmt{opt1},\dots,
\fmt{opt9}. A smaller number encourages line breaking at this point more
than a higher number. So \fmt{opt0} will nearly always lead to a line
break whereas \fmt{opt9} leads to a line break only if there is no
\fmt{opt0},\dots, \fmt{opt8} primitive somewhere near and the line {\em
has} to be broken around this point.\footnote{In \TeX\ terms the digit
$n$ in the name of the \fmt{opt} command specifies the penalty
associated with this break point. The penalty in the default
interpretation of \fmt{opt} is about $10\cdot n$.} So the \fmt{opt}
command denotes an optional line break with the continuation line
indented a little more with respect to the normal indentation.

\index{opt}
\index{format commands!opt}

So in the example we therefore say:
 %
\begin{inp}
while (str\_buf[str\_buf\_ptr]<>c) and \fmt{opt4}
(str\_buf\_ptr<=buf\_size) do \fmt{indent} \fmt{force}
begin ...\fmt{force}
end
\end{inp}
 %
depending on whether we rather liked a line break before or after
the ``\t{and}''. This would then yield the correct way of formatting,
as in:
 %
\begin{outp}
\res{while} $(\id{str\_buf}[\id{str\_buf\_ptr}]<>\id{c})$ \andsign
\myindent\myindent\myindent\myforce
$(\id{str\_buf\_ptr}\leq\id{buf\_size})$ \res{do}\myoutdent\myoutdent\myforce
\res{begin}\dots\myforce
\res{end}
\end{outp}
 %
Note however that a \fmt{break\_space} is always
a better place to break a line than an optional line break denoted
by \fmt{opt}.

\subsubsection*{Additional Commands}

\index{format commands!additional}

The list of format commands is not yet complete. There
are still a few more primitives to learn, but they are not as
important as the other commands that were presented up to this
point. The following primitives were introduced to achieve better
formatting results in special situations.

\index{backup} \index{format commands!backup}

The first one is called \fmt{backup} and denotes a little negative
horizontal space. This command must be preceded by a
\fmt{force}, because it doesn't make sense to backup within a line
if it's beginning has already been output. So the two commands
``\fmt{force}, \fmt{backup}'' appearing in a row can be seen as a
short form for the sequence ``\fmt{outdent}, \fmt{force},
\fmt{indent}''. The \fmt{backup} command is used in cases where
statements are preceded by labels or simply where a line should
stick out to the left a bit in order to emphasize it.
 %
\index{Format commands!backup@\fmt{backup}}
\index{backup@\fmt{backup}}


The next additional command is called \fmt{big\_force}. As the
name implies this primitive acts like a \fmt{force}, but on a
bigger scale. \fmt{big\_force} not only forces a line break; it
also inserts a little extra vertical space after the break. You
can use the \fmt{big\_force} primitive to separate two adjacent
lines that semantically do not belong together, like two function
declarations.
 %
\index{Format commands!big force@\fmt{big\_force}}
\index{big force@\fmt{big\_force}}


If you are trying to typeset \C\ or \CPP\ programs, you will need
a feature to typeset so called preprocessor directives. These are
commands beginning with a $\#$ that can influence the source code
before compilation. Normally these lines appear beginning at the
left margin so we want a command that can be used to typeset lines
like this. This command is called \fmt{no\_indent}. It causes the
{\bf current} line to be printed flushleft. So if you want to
prettyprint a \C\ or \CPP\ preprocessor line, you should say
 %
\index{c++@\CPP}
\index{c@\C}
\index{Preprocessor}
\index{c preprocessor@\C\ preprocessor}
\index{Format commands!no indent@\fmt{no\_indent}}
\index{no indent@\fmt{no\_indent}}
 %
\begin{inp}
\fmt{force} \fmt{no\_indent} $\#$PREPROCESSOR\_DIRECTIVE
\fmt{force}
\end{inp}
 %
and the line following this directive will be indented as much as
the line before the directive.

%cancel

Finally the last command of the present command set is called
\fmt{cancel}. To understand this command I have to mention a tricky
detail of the presented algorithm that I have omitted in the
intoduction of this section: There I said that format commands are
simply output by the prettyprinter one by one just like strings, only
that they are ``marked'' as typesetting commands.  Actually the
prettyprinter also does a little bit of interpretation in advance. But
these exceptions from the rule can be caught in two
phrases\label{two-phrases}. The first one is of global importance and
the second one is related to the \fmt{cancel} command:

\index{Format commands!cancel@\fmt{cancel}}
\index{cancel@\fmt{cancel}}
\index{Exceptions}
\index{Preprocessing of format commands}
\index{Tricky details}

\begin{enumerate}

\item A sequence of consecutive \fmt{break\_space}, \fmt{force},
and/or \fmt{big\_force} commands is replaced by a single command
(the maximum of the given ones).

\item The \fmt{cancel} command cancels any \fmt{break\_space},
\fmt{opt}, \fmt{force} or \fmt{big\_force} command that immediatly
precede or follow it and also cancels any \fmt{backup} command
that follows it.

\end{enumerate}

So it seems that the \fmt{cancel} command is a command that mixes
everything up again after we have put everything into a neat and tidy
form! But this only seems so. The interpretation done by the
prettyprinter according to the two rules stated above, was introduced
to be more generous and more flexible with the placement of format
primitives into the source code. Because, after all, the format
commands will not be inserted by hand (as the reader might think after
viewing the examples given above), but they will be inserted
automatically by the prettyprinter according to the prettyprinting
grammar that we have been introduced to above.
 %
\index{Flexibility}
\index{Prettyprinting grammar}
\index{Format commands!explained|)}


\subsubsection*{Summary of Format Commands}

\index{Format commands!summary}
\index{Summary of format commands}

Table \ref{command-table} shows a complete list of all the format command
primitives introduced in this part of the book. The only command that
wasn't mentioned up to now is the \fmt{null} primitive that serves
as a neutral (or empty) command. Now, with the knowledge of all the
intrinsic and extrinsiv features of the prettyprinting method of
\pretzel{} prettyprinters, you are ready to read, write or modify
prettyprinting grammars at your will. Now, you have total control
over the prettyprinting style; fully user controlled prettyprinting
is in your grasp.

\index{Format commands!null@\fmt{null}}
\index{null@\fmt{null}}

\begin{figure}[htb]
\begin{center}
\begin{tabular}{lp{7cm}}
\fmt{null}    & empty command \\
\fmt{indent}  & indents the next line a little more \\
\fmt{outdent} & takes back the last indentation (de-indent) \\
\fmt{force}   & forces a line break \\
\fmt{break\_space} & denotes a possible space for a line break \\
\fmt{opt$\,n$}, $n \in \{0,\dots,9\}$ & denotes an optional line
     break with the continuation line indented a litte with
     respect to the normal starting position. This line break will
     score a penalty of $10n$.\\
\fmt{backup} & denotes a small backspace\\
\fmt{big\_force} & forces a line break and inserts a little extra
     space \\
\fmt{no\_indent} & causes the current line to be output
     flushleft\\
\fmt{cancel} & obliterates any \fmt{break\_space}, \fmt{opt},
     \fmt{force} or \fmt{big\_force} command that immediatly
     precedes or follows it and also cancels any \fmt{backup}
     command that follows it.
\end{tabular}
\end{center}
\caption{List of format command primitives}
\label{command-table}
\end{figure}





\section{Writing Prettyprinting Grammars}

\begin{quote}

  ``Knowing \CWEB{} and its sources, it is clear to me that doing a good
  job (or even a fairly good job) about prettyprinting is not a simple
  task.''

  Marc van Leeuwen \cite{vanLeeuwen96:pp:noweb}

\end{quote}

\index{Leeuwen@van Leeuwen, Marc}

\noindent Yes, this is true. In terms of \pretzel{}, ``doing a good job
about prettyprinting'' is writing a good prettyprinting grammar. For
simple languages you might spend a day or two preparing a good grammar,
but for more complex languages (and more complex wishes to the look of
the output) you might spend weeks and still not be happy.

\index{prettyprinting!grammars}
\index{grammars!prettyprinting}
\index{complex languages}


\subsection{Modifying an existing grammar}

If you're lucky, somebody else had done the work already for you. In
such cases, simply grab a copy of the two \pretzel{} input files, run
\t{pretzel-it} and off you go. The \pretzel{} homepage
\cite{pretzel:homepage} should be a good starting point for the quest
for a prettyprinting grammar for your language.

\index{Pretzel!homepage} \index{homepage!Pretzel}

But what if this grammar doesn't suit your preferences? What if it
prettyprints ugly code? Well, then you have to plunge into the files
and modify them. Making small changes that will work is quite easy.
For example, if you only dislike the way certain symbols come out of
the prettyprinter, look in the formatted token file and try to figure
out, where you can change it. Or if you dislike forced line breaks at
certain places, just go into the grammar and remove the \fmt{force}
from the attribute definition.

If your wishes cover more major aspects of the layout, then you should
make a copy of the file and rename it to something that reflects the
change. Also, you might put a new comment at the beginning of the file
that describes who you are and what this new grammar does differently
than the grammar you started from. Now you can start to experiment.
The good thing about this is that you can always fall back to the
original copy and start from there again. If you get your new grammar
working, contact the \pretzel{} home page \cite{pretzel:homepage} so
it can be added to the list of files for certain languages. Other
people can then use your grammar and don't have to go through the same
work again.

For similar languages (like Java and \C{}) you could try to take a
grammar for the first one and try to modify it that it fits the second
one. In any case, studying existing grammars is always very helpful if
you don't have much experience.

\index{Java} \index{C}


\subsection{Writing a new Grammar from Scratch}

If you're unlucky, and you can't find a suitable grammar then you will
have to start from scratch. If you're in this situation, there are two
possible ways of continuing:
 %
\index{writing grammars} \index{scratch}
 %
\begin{enumerate}

\item Most programming languages are context free and so there should be
a context free grammar for your language available either in  the
reference works (i.e.~the books) that ``define'' the language, or
somewhere in the internet, where there could be compilers for your
language available in the source, and if so, a context free grammar for
the parser of that compiler is almost always included there. You could
start with such a grammar and simply add formatting instructions to the
rules as you wish.

\index{language definition grammars} \index{reference grammars} \index{compilers}
\index{books}

\item You could start entirely from scratch, look at the language and start
writing a prettyprinting grammar that works with the code you are working
on. This might seem more work than the first alternative, but sometimes
is still feasible.

\end{enumerate}

To go with the first alternative, you will have to have a full blown
language grammar available.  But these grammars (as for example with
\C{} or \CPP{}) tend to be rather large and ugly, resulting in
prettyprinting grammars, that are difficult to understand and  difficult
to modify. However, a prettyprinting grammar doesn't  have to be a full
blown language grammar after all. The fact that such a grammar will
solemnly be used to create a ``prettyprinted'' translation of a bit of
source code it doesn't necesserily have to have any resemblence to a
language grammar. In \PASCAL{} for instance, \res{for}-, \res{while}-
and \res{else}less \res{if}-statements are normally formatted in a very
similar way; so the grammar mustn't distinguish between the three
constructs. This results in prettyprinting grammars that can be quite
different from the normal grammars of the programming languages. In most
cases they are much smaller.

\index{prettyprinting grammars}

However, if you start from scratch and don't want to use an existing
reference grammar, you could still consult it once and again if you are
stuck with your own one. Like the language grammar, the prettyprinting
grammar must be able to parse {\em every} construct that will appear in
the input. But it's no problem if the prettyprinting grammar might
swallow code that isn't syntactically correct. We're talking
Prettyprinters here and not Compilers. Sections
\ref{sec:context:sensitive} and \ref{sec:debugging} will elaborate more
on the issue of ``grammar correctness.'' In any case, having a look
at available prettyprinting grammars in advance helps a lot.

\index{pascal@\PASCAL}



\subsection{Context Free versus Context Sensitive}
\label{sec:context:sensitive}

\index{context free grammars} \index{context sensitive grammars}

Looking at the history of the prettyprinting method that \pretzel{}
employs (see chapter \ref{chap:on:prettyprinting}), we can see that the
grammars that the first systems started with were context sensitive.
This means that there actually are ready to use context sensitive
prettyprinting grammars around for many languages such as \PASCAL{}
\cite{Knuth:web83} and \C{}/\CPP{} \cite{Knuth:CSS93}. There is even a
system similar to \pretzel{} but with a more complete approach called
{\sc SPIDER} \cite{Ramsey:CACM-32-9-1051} that amoung other things
created a prettyprinter from a context sensitive prettyprinting
grammar.

\index{SPIDER}
\index{WEB}

So why does \pretzel{} use context free grammars anyway? The answer
to this problem lies in the tools that \pretzel{} relies on. The
prettyprinters that \pretzel{} generates are actually parsers, and
instead of writing a whole new parser generator, \pretzel{} relies
on the well known \UNIX{} tool Bison, that does just the work we need.
But, in effect, Bison input is context free! Hmmm.

\index{Bison} \index{parser generator} \index{UNIX}
\index{Hmmm}

Now context free grammars are a little more restrictive than context
free ones, and so it is not possible to transform any context sensitive
grammar of your choice into a context free one. This is unfortunate,
but this is a fact we have to live with. My knowledge of formal language
theory is not very deep, but I suspect that parsing with context
sensitive grammars is not trivial and that this is the reason why
no generators like Bison exist in a context sensitive flavour.

\index{formal language theory} \index{parsing} \index{Bison}

So context free is what we have to live with. From studying existing
prettyprinting systems (again see chapter \ref{chap:on:prettyprinting}
for details) it seems that context sensitive is the way to go in
prettyprinting, and this seems bad for \pretzel{}.\footnote{I actually
  {\em is} a quite severe restriction and will probably be the reason
  why \pretzel{} will not be widely used in practice. The extension of
  being able to add \C{} code to the attribute definition only
  partially eases the situation (see section \ref{sec:adding:code} for
  details).} But \pretzel{} is a prototype and (at the moment) the
best you can get as a prettyprinter generator and a context sensitive
implementation of \pretzel{} is left for a following student
generation to pursue. The good thing about context free grammars
however is that every context free grammar is also a context sensitive
grammar, and so --- if a context sensitive version of \pretzel{} is
ever built --- every existing and working prettyprinting grammar can
be used with the new system without a single change. At least this is
good news. For more information on this topic, also see chapter
\ref{chap:future:work} on ``Future Work.''

\index{restrictions of Pretzel} \index{good news}


\subsection{Available Grammars}

All those prettyprinting grammars that are available in \pretzel{} format
are contained in the latest distributions of \pretzel{} (at least all
that I know of). If you have a good grammar for your favourite
programming language at hand, let me know and I'll include it in the
distribution too.

\index{available grammars}
\index{prettyprinting grammars!available}

All these input files are available individually from the \pretzel{}
homepage \cite{pretzel:homepage}. So if your \pretzel{} installation
is rather old, then surf to that URL and look if there's something new
on the page.

To date (July 9, 1997) I am aware of:
 %
\begin{itemize}
\item \C{} (by Felix G\"artner) \\ Directory: \t{languages/cee} \\
  Files: \t{cee.ft} and \t{cee.fg}

\index{Gaertner@G\"artner, Felix}

\item \PASCAL{} (by Felix G\"artner) \\ Directory:
  \t{languages/pascal} \\ Files: \t{pascal.ft} and \t{pascal.fg}
\item Java (for \noweb{} by Lee Wittenberg) \\ Directory:
  \t{contrib/leew} \\ Files: \t{java.ft} and \t{java.fg}

\index{Wittenberg, Lee}

\item Java (for \noweb{} by Felix G\"artner) \\
  Directory: \t{contrib/noweb/java.latex} \\ Files: \t{javafx.ft} and
  \t{javafx.fg} \\ Comments: Based on the grammar by Lee Wittenberg
  with slight changes, but enhanced to do indexing.
\item Java (for \noweb{} by Holger Uhr) \\
  Directory: \t{contrib/huhr} \\ File: \t{javahu.fg} \\ Comments:
  Replaces \t{java.fg} from Lee Wittenbergs version in \t{contrib/leew}.

\index{Uhr, Holger}

\end{itemize}
 %
Always see the \t{README} files in the directories for details.

\index{gaertner@G\"{a}rtner, Felix}
\index{Wittenberg, Lee}
\index{Java} \index{Pascal} \index{C}


\subsection{Debugging Prettyprinting Grammars}
\label{sec:debugging}

\index{debugging grammars}

\begin{quote}

  ``Rather, it [the need to type `\verb"@;"' after each refinement] is
  caused by an error made in converting \WEB{} to \CWEB{} that was
  never corrected, namely a failure to recognise the fundamentally
  different roles of semicolons in \PASCAL{} and \C{}.  In \PASCAL{}
  semicolons separate statements, whether simple (e.g., assignments)
  or compound, so if any such statement is followed sequentially by
  another one, a semicolon is required after the first; the
  requirement still holds if the first statement is a refinement. In
  other words, in Pascal one has `refinement $\rightarrow$ statement'
  always, without having to consider \verb"LINE_END", and without
  having to use `\verb"@;"' (either a real semicolon is required after
  it, or nothing is, e.g., when \t{ELSE} follows). In \C{} however, no
  symbol separates successive statements, but things like assignments
  are expressions rather that statements, and they only become
  statements by incorporating a following semicolon. Since refinements
  almost always stand for (compound) statements, they should be given
  that category; however in (Levy/Knuth) \CWEB{} they are given the
  same category as assignments, which is `expression'. This forces one
  to write a semicolon after each refinement, even though that really
  becomes a spurious empty statement; if such an empty statement would
  mess up the real syntax (the compiler would fail) then one has to
  write `\verb"@;"' instead.''

  Marc van Leeuwen \cite{vanLeeuwen95:pp:layout:style}

\end{quote}

\index{Leeuwen@van Leeuwen, Marc}


\noindent Almost certainly, the first experiences with your own (or
other's) prettyprinting grammars will lead to situations where source
code is not prettyprinted the way you wanted and you don't know why. The
insight that follows from this situation is very frustrating: nothing
comes for free! Searching for  bugs in prettyprinting grammars is as
necessary as in normal programs.

\subsubsection*{The \t{error} Token}

If a \pretzel{} prettyprinter can't handle a sequence of input code,
it will output {\em nothing}, not even an error message. But there is
still a way to notice this fact. There is a special \t{error} token
that may be used in such circumstances. You can place it into your
grammar at any place, and it will match a sequence of tokens not
covered by the other rules in question.

\index{error token} \index{robust grammars}

A handy place to put an \t{error} token is right at the beginning
of the grammar. Say, a \t{final} token is the token that everything
reduces to in your grammar. To notice a parse error, you can for
example write
 %
\begin{verbatim}
final   : exp
        | stmt
  // check for parse errors:
        | error  { "syntax error" }
;
\end{verbatim}
 %
and so the text ``syntax error'' appears in the output everywhere
\pretzel{} wasn't able to parse the input.

\index{syntax error}

You may place the \t{error} token in any place you like, as long as
it is on the right hand side of the colon. Using the \t{error} token
in other places than at the end, you may produce grammars that are
more robust and can handle unforseen constructs more flexibly.


\subsubsection*{Watching the Parse}

\index{watching the parse}
\index{prettyprinting grammar!watching parse}

The \t{pretzel-it} program offers a \t{-d} option that produces a
prettyprinter which is in ``debugging mode'' by default. This means
that during prettyprinting, the program will output a lot of debugging
information to the standard error stream (usually the screen). This
information displays what kind of token has been read from the
input (and how it looks) and what rule of the prettyprinting grammar
has been chosen. Additionally, the \t{-d} option produces a file
that contains an analysis of your prettyprinting grammar. If your
input name is, say, \t{foo}, then this file will be called \t{foo.output}.

\index{pretzelit@\t{pretzel-it}!option \t{-d}}
\index{d@\t{-d} option of \t{pretzel-it}}
\index{debugging mode}

Here's an example of the debugging output (taken from a run
on the \t{small-example.simpas} example):
 %
\begin{verbatim}
Starting parse
Entering state 0
Reading a token: Next token is 260 (IFif)
Shifting token 260 (IF), Entering state 1
Reading a token: Next token is 266 (IDa)
Shifting token 266 (ID), Entering state 4
Reducing via rule 7 (line 195), ID  -> exp
state stack now 0 1
Entering state 8
Reading a token: Next token is 259 (BINOP=)
Shifting token 259 (BINOP), Entering state 11
Reading a token: Next token is 265 (NUM1)
Shifting token 265 (NUM), Entering state 3
Reducing via rule 8 (line 196), NUM  -> exp
state stack now 0 1 8 11
Entering state 15
Reading a token: Next token is 261 (THENthen)
Reducing via rule 9 (line 197), exp BINOP exp  -> exp
[...]
Entering state 14
Reducing via rule 2 (line 102), stmt_list SEMI stmt  -> stmt_list
state stack now 0
Entering state 5
Now at end of input.
Shifting token 0 ($), Entering state 19
Now at end of input.
\end{verbatim}
 %
Every time, the scanner reads a token, the token code (like \t{IF}
or \t{ID}) is printed which is followed by the actual contents of the
token (like ``\t{a}'' in case of the identifier). Every time the
parser reduces a token, it prints the rule that it uses. The number
of the rules and also the state information will become clear if you
look at the \t{simpas.output} file and learn about the internals of
the \pretzel{} parsing mechanism (by looking at the manual page
\t{bison(1)}).


\subsubsection*{Context Free again}

As described above, context free grammars are pretty restrictive,
sometimes even more restrictive than we like. Note that rules like
 %
\begin{verbatim}
decl_head id :   int_like id
;
\end{verbatim}
 %
to turn an \t{int\_like} token into the head of a declaration if it
is followed by an identifier are context senstive and therefore forbidden.

\index{context sensitive grammars}
\index{restrictions}

See also the next chapter, especially section \ref{sec:tips:and:tricks}
for a few more tips and tricks for writing and debugging prettyprinting
grammars.


\subsection{Experiences}

\begin{quote}

  ``With \CWEB, there'll always be something someone wants and can't
  have, because it tries to do too much in its kernel.  Witness the
  recent complaint in this newsgroup from someone trying to put
  makefiles in the \CWEB{} sources.  \CWEAVE{} tries to be the perfect
  \C{} programming tool right out of the box, and so comes up
  completely unable to handle makefiles.''

  Barry Schwartz \cite{schwartz94:use:of:prettyprinting}

\end{quote}

\index{Schwartz, Barry}

\noindent Remember, writing good prettyprinting grammars is difficult.
So don't overdo it. It's pretty difficult (if not impossible) to come
out with a ``complete'' grammar that will handle all cases brilliantly.
So if you're happy with the way your code looks, that's fine. If
something turns out wrongly, ask yourself, what actually {\em is} wrong
here?







% ######################################################################

\chapter{\pretzel{} Hacking}
\label{chap:pretzel:hacking}

In this chapter we'll have a look into the guts of \pretzel{}. We'll
see how the internals of \t{pretzel-it} work and what other nice
things you can do with the system. This chapter is aimed at
programmers who want to produce simple, efficient and elegant
prettyprinters for their language. However, I recommend reading at
least the first subsection about adding \C{} code to the rules, even
when you dislike hacking.


\section{Adding \C{} Code to the Rules}
\label{sec:adding:code}

\index{adding code}
\index{C code in rules}

Yes, you cann add your own \C{} code to the attribute definition parts
of the formatted grammar and formatted token files. A typical
application of this feature is to dynamically typeset constructed type
names (as \res{typedef}s in \C{} or \res{class}es in Java) correctly.

The problem with \pretzel{} is, that we would like to perform
different actions depending on the context of the token that the
prettyprinter finds. For example, in \C{} normal identifiers should be
typeset in italics, but identifiers that are the names of
\res{typedef}s should be typeset in bold. \pretzel{} can't handle this
situation without tricky manipulations of the attribute definitions,
because the type of grammar that \pretzel{} uses is essentially
context free (see section \ref{sec:context:sensitive} for a
discussion of related problems).

\subsection{Example for Tokens}

How handle this situation? \pretzel{} allows you to add \C{} code to
your attribute definitions. Here is a small example, how this could be
used in formatted token files:

\begin{verbatim}
[a-zA-Z][a-zA-Z0-9_]*    ID  { "{\\it "  [escaped_underlines(yytext)] "}" }
\end{verbatim}

The purpose of this line is to scan tokens that are identifier names
and return ``\t{ID}'' type tokens to the prettyprinting parser, but
also to modify the appearance of the matched text before returning. In
this example there is \C{} code inside the attribute definition. This
code is encircled in angled brackets ``\t{[}'' and ``\t{]}'' and can
be any sequence of \C{} statements with one central property: The code
part must reduce to the \res{Attribute}$\ast$ class, i.e.~the return type
must me a pointer to an \res{Attribute} object. The \res{Attribute} class
is a special \CPP{} class that belongs to the \pretzel{} system and
is contained in the runtime library.\footnote{A description of how to use
objects of this class and related functions can be found in the
\noweb{} file \t{attr.nw} in the directory \t{attr} of the \pretzel{}
distribution.}

\index{[, code delimiters}
\index{code delimiters}
\index{Attribute class}
\index{attr@\t{attr.nw}}

% attr.nw might also belong into this book? XXX

To this effect, the {\it escaped\_underlines} function called above
must have the syonpsis:
 %
\begin{outp}
\res{Attribute}$\ast$ \id{escaped\_underlines}(\res{char}$\ast$ \id{s});
\end{outp}
 %
The matched text is accessible via a string (i.e.~a \res{char}$\ast$
variable) called \id{yytext}. The delimiters of code are angled brackets
as explained above. If angled brackets appear in the \C{} code itself,
any closing bracket must be escaped by a backslash in order for \pretzel{}
to recognize it correctly.

\index{escaped underlines}
\index{yytext}


To help simplify the synopsis of the \id{escaped\_underlines} function,
we may use special functions in the runtime library that belong to the
\res{Attribute} class. These are the two functions \id{create} and
\id{join}. The \id{create} function takes a string (i.e~a
\res{char}$\ast$) and turns it into an attribute. So, if you have
written a function, say \id{esc\_ul}, that turns a string into the same
string with escaped underlines, then an implementation of the function
above could look like this:
 %
\index{create} \index{join}
 %
\begin{outp}
\res{Attribute}$\ast$ \id{escaped\_underlines}(\res{char}$\ast$ \id{s})
\myforce\{\myindent\myforce\res{return} \id{create}(\id{esc\_ul}(\id{s}));
\myoutdent\myforce\}\myforce
\end{outp}
 %
The \id{join} function simply joins two or more (up to ten) attributes
together into an new one.

\subsection{Example for Grammars}

But you can do more with this feature. Here's an example code fragment
that handles the ``correct'' formatting of \res{typedef}s in \C{}. In
\C{} you can construct your own shorthands for types by using the
\res{typedef} construct, thus making normal identifiers like
\id{Length} to type names. Here's an example, if you're not aquainted
with \C{}:
 %
\begin{outp}
  \res{typedef} \res{int} \id{Length};\pretzelbigforce
  \id{Length} \id{l};\pretzelforce
\end{outp}
 %
The identifier \id{Length} has the status of a type name and this
should be reflected in the typesetting style. So the way we want it
would be like this:
 %
\begin{outp}
  \res{typedef} \res{int} \res{Length};\pretzelbigforce
  \res{Length} \id{l};\pretzelforce
\end{outp}

The idea behind a simple solution to this problem is to use a simple
lookup table to keep track of all identifiers that have been
\res{typedef}ed. Then, if we have to prettyprint an identifier, we
simply look into the table and from the result we know whether we can
use boldface or italics.

A prettyprinter for \C{}\footnote{Like the one in the {\tt
    languages/cee} subdirectory of the \pretzel{} distribution.} might
have rules like these to handle \res{typedef} statements:
 %
\begin{verbatim}
typedef : TYPEDEF_LIKE INT_LIKE ID      [ install($3); ]
                                        { $1 "\\ " $2 "\\ {\\bf " $3 "}" }
;
\end{verbatim}
 %
(We assume that \id{install} is a function that installs an identifier
in the lookup table and that \id{lookup} is a function that tells us
if an identifier is in the table.)

\index{install} \index{lookup table}

What we see here is called {\em starting code}. Starting code is code
in angled brackets that is placed before the attribute definition. It
can contain any sequence of \C{} code that you like and will be
executed before the attribute of the reduced token is even touched.
You can use it to place declarations of variables or any other actions
that should take place before the attribute is pieced together.

So here, in this rule, we install an identifier that is
\res{typedef}ed into the table. The identifier is typeset in bold
already (but you could put it into any font you like). Now, every time
we handle an identifier within declarations or expressions, we lookup
it's name in the table. A rule like this does the job:
 %
\begin{verbatim}
id      : ID            { [lookup($1) ? create("{\\bf ") :
                             create("{\\it ")] $1 "}" }
;
\end{verbatim}
 %
Note that we are making use of the conditional expression of \C{}.
Remember that the code within the attribute definition has to turn out
as type \res{Attribute}$\ast$ and so we use the \id{create} function
again to turn a string into an attribute. So here again we use code {\em
within} attribute definitions and all in all, we can use identifiers as
\t{id} tokens in the grammar and know that they are typeset correctly.

\index{starting code} \index{ending code}

Apart from code before the attribute definition and code within
attribute definitions, there can also be code after the attribute
definition which is called {\em ending code}. Ending code can be any
kind of sequence of \C{} statements. They will be executed {\em after}
the attribute definition has been put together and {\em before} the
final token identifier id returned. So in effect you yourself may play
God and return a token identifier yourself. The following extract from
a formatted token file is an example:
 %
\begin{verbatim}
[a-zA-Z][a-zA-Z0-9]*      ID    { ** }    [ return(ID); ]
\end{verbatim}
 %
Here, the \res{return} statement is executed just before the
prettyprinting scanner itself returns the \t{ID} token, so here you
could leave the ending code away and nothing would be changed. But of
course, you could put an \res{if} statement there and return something
else as you wish. The symbolic names of all the tokens may be accessed
with the code parts simply by their names you gave them.

Now you might wonder, what the sequence
 %
\begin{verbatim}
[a-zA-Z][a-zA-Z0-9]*      ID      [ return(ID); ]
\end{verbatim}
 %
would mean. Tricky you! Is this starting or ending code?  Well,
\pretzel{} doesn't want to care about these kind of questions, and so
it simply declares them illegal. If you want to use either starting
code, ending code or both, you {\em need} an attribute definition.



\subsection{Summary}

This is how \pretzel{} expects you to write code inside your
\pretzel{} input files:
 %
\index{code in attributes!summary}
 %
\begin{itemize}
\item Code fragments are bracketed within angled brackets. Any angled
  brackets that appear within the \C{} code must be escaped with a
  backslash.
\item Starting code and ending code are written outside of the
  attribute definition. Both are totally optional, but if you want to
  specify either or, you need an attribute definition.
\item Code parts within attribute definitions must return a pointer to
  an \res{Attribute} class object.
\item Within the formatted token file, the matched text is visible
  to you in form of a \res{char}$\ast$ variable called \id{yytext}. The
  symbolic names of the tokens are available by the same name that
  \pretzel{} gives them.
\item Starting code, code within attribute definitions and ending code
  is totally optional. But at any place where they are allowed, only
  one bracketed code bit may be placed.
\end{itemize}
 %
Common routines to escape identifiers, to build and manage lookup
tables, to convert to and from \res{Attribute}$\ast$ or to output debug
information can be found in the files belonging to the \C{}
prettyprinter in the directory {\tt
  languages/cee} of the \pretzel{} distribution.




\subsection{Tips and Tricks}
\label{sec:tips:and:tricks}

\index{tips and tricks}

One thing I have found usefull is to slip debug output into the
\pretzel{} input files via \C{} code added to the attribute
definitions. For example, I have written a function called
\id{debug\_print} that takes a string and an \res{Attribute}$\ast$ and
outputs the contents of the attribute to \id{stderr}. Here is what it
could look like:
 %
\begin{outp}
{\bf static} {\bf void} {\it debug\_print}({\bf char}$\ast$ {\it this\_is},
\pretzelopt9{}{\bf Attribute}$\ast$ {\it stuff})\pretzelforce{}\{
\pretzelindent{}\pretzelforce{}{\it cerr} $\ll$ \verb*|"*** "| $\ll$
{\it this\_is} $\ll$ \verb*|":"| $\ll$ {\it endl};\pretzelforce{}
{\bf Latex\_cweb\_output} {\it os}({\it cerr});\pretzelforce{}
{\it stuff}$\to${\it print}({\it os});\pretzeloutdent{}\pretzelforce{}\}
\pretzelforce{}
\end{outp}
 %
\index{debug print}
\index{Latex cweb output}
 %
This will use a \res{Latex\_cweb\_output} object (see the file
\t{output.nw} in the \t{output} directory of the \pretzel{} distribution
or section \ref{sec:non:latex} for details) to output the contents of an
attribute to the standard error stream. The idea now is to insert a code
fragment that outputs the contents of an attribute at special places in
the grammar, like for example before the prettyprinter returns:
 %
\begin{verbatim}
  final : exp           { "$" $1 "$" }  [ debug_print("Final exp", $1); ]
\end{verbatim}
 %
This will output the contents of the expression \t{exp} to the
standard error stream before reducing it to a \t{final} token and
exiting.

Also, \pretzel{} prettyprinters do not insert newlines into the output
if they are not told to. So, if no special care is taken, most
prettyprinters will output one single long line of prettyprinted
code. This might upset some typesetting systems that have restricted
input buffers and are used to handle one line at a time. This problem
may be avoided by manually inserting newline characters from time to
time into the output, as for example after every statement:
 %
\begin{verbatim}
stmt : exp semi      { $1 $2 force "\n" }
\end{verbatim}
 %
This will lead to better readable input and will make a lot of
formatting programs happier.

\index{newlines}

There are a couple of handy functions available that do stuff like
escaping underlines and turning an attribute into a string. Look at
the files \t{javafx.ft} and \t{javafx.fg} in the
\t{contrib/noweb/java.latex} directory of the \pretzel{} distribution
for details. They are self-contained \pretzel{} input files that have
a hash table built into them and do all converting by themselves.


\section{The \pretzel{} Interface}
\label{sec:pretzel:interface}

\index{Pretzel!interface}

The actual \pretzel{} program (called \t{pretzel}) does only half the
job of \t{pretzel-it}. It simply builds ready to use source code for
the specified prettyprinter in terms of input files to the two
standard \UNIX{} tools flex and Bison. Using these tools will result
in compilable \CPP{} source code that contains the actual
prettyprinting module.

\pretzel{} prettyprinters consist of two parts:
 %
\begin{enumerate}
\item a {\em prettyprinting scanner} that cuts the input into small
  pieces (called {\em tokens}), and
\item a {\em prettyprinting parser} that joins the tokens returned by
  the scanner together and does the actual prettyprinting.
\end{enumerate}
 %
Both, the scanner and the parser, are seperate modules with a well
defined interface. In this section I will present this interface.

\index{prettyprinting scanner}
\index{prettyprinting parser}

\subsection{The Prettyprinting Scanner}

The prettyprinting scanner is a \CPP{} prettyprinting scanner class.
Every prettyprinting scanner produced by \pretzel{} is a subclass of
an abstract scanner class that looks like this:
 %
\index{prettyprinting scanner class}
 %
 % why isn't this prettyprinted? XXX
 %
\begin{verbatim}
#include<iostream>
#include"attr.h"

class Pscan {
public:
  Pscan(istream*) {};
  ~Pscan() {};
  virtual int scan(Attribute**) = 0;
};
\end{verbatim}
 %
A \pretzel{} scanner must be associated with an \CPP{} input stream
object that is passed to the constructor of the class. The scanner
will read characters from this input stream when scanning. The actual
scanning if performed inside the \id{scan} member function. It's
return codes are integers and identify the token that it has just
scanned. If the input is empty, it returns 0.

\index{Pscan class}

The subclass that \pretzel{} produces will always be a subclass of the
above class. It's interface is similar:
 %
 % prettyprint this XXX
 %
\begin{verbatim}
class PSCAN_NAME : public Pscan {
public:
  PSCAN_NAME(istream*);
  ~PSCAN_NAME();
  int scan(Attribute**);
};
\end{verbatim}
 %
The name of the generated class is \t{PSCAN\_NAME} by default and has
to be redefined using preprocessor macros, such as for example
 %
\index{pscanname@\t{PSCAN\_NAME} macro}
 %
\begin{verbatim}
#define PSCAN_NAME Pscan_for_pascal
\end{verbatim}
 %
By default the name of the derived subclass will be \t{Ppscan}.
A header file containing the derived scanner class could look like
this (by default the header file is called {\tt Ppscan.h}). Note that
there is no precaution against double inclusion.
 %
\index{pscanh@\t{Pscan.h} header file}
 %
\begin{verbatim}
/* header file for a prettyprinting scanner */

/* NB: This file is NOT protected against double inclusion! */

#include "Pscan.h" // include abstract base class

#ifndef PSCAN_NAME
#define PSCAN_NAME Ppscan
#endif

class PSCAN_NAME : public Pscan {
public:
  PSCAN_NAME(istream*);
  ~PSCAN_NAME();
  int scan(Attribute**);
};
\end{verbatim}
 %
This is exactly the default header file that you can find in the
\t{include} directory of the \pretzel{} distribution. If \pretzel{} is
installed on your system, you should find this file (and the header
file for the abstract base class) in the global \pretzel{} include
directory too.\footnote{This directory will usually be called
  \t{/usr/local/lib/pretzel/include} or something alike.}


\subsection{The Prettyprinting Parser}

\index{prettyprinting parser interface}

The interface to the prettyprinting parser is similar in structure to
the prettyprinting scanner. We have an abstract base class called
\t{Pparse} that has this interface:
 %
\index{pparse@\t{Pparse} class}
 %
\begin{verbatim}
#include<iostream>
#include"attr.h"
#include"output.h"

class Pparse {
public:
  Pparse() {};
  ~Pparse() {};
  virtual int prettyprint(istream*, ostream*) = 0;
  virtual int prettyprint(istream*, Output*) = 0;
};
\end{verbatim}
 %
There are the usual functions to create and destroy a prettyprinting
parser plus an overloaded virtual member function \id{prettyprint}.
The normal use of \id{prettyprint} uses \CPP{} input and output
streams. It reads from an input streams and writes prettyprinted code
to the output stream.

\index{prettyprint function}

The second version of this member has an \id{Output} class pointer as
second argument. This is a version for experts. See section
\ref{sec:non:latex} for an example how to utilize this.

The prettyprinting parser class which is actually generated is again a
subclass of this abstract base class. It's name may be redefined using
\C{} preprocessor macros.
 %
\index{pparsename@\t{PPARSE\_NAME} macro}
 %
\begin{verbatim}
class PPARSE_NAME : public Pparse {
public:
  PPARSE_NAME();
  ~PPARSE_NAME();
  int prettyprint(istream*, ostream*);
  int prettyprint(istream*, Output*);

  void debug_on();
  void debug_off();
};
\end{verbatim}
 %
The two extra functions \id{debug\_on} and \id{debug\_off} enable and
disable debugging output. Debugging output gives an indication what
the scanner returns and what kind of rules the parser is reducing. It
is really a lot of information.\footnote{The information provided
using the \t{-d} option of \t{pretzel-it} program comes from using
these functions (see section \ref{sec:debugging}).}

\index{debug on function} \index{debug off function}
\index{d@\t{-d} option of \t{pretzel-it}}

As you can see, the name of the generated class is \t{PPARSE\_NAME} by
default and has to be redefined using preprocessor macros, such as for
example
 %
\begin{verbatim}
  #define PPARSE_NAME Prettyprinter_for_pascal
\end{verbatim}
 %
By default the name of the derived subclass will be defined as
\t{Ppparse} (note the three `p's). This is the default header file for
the generated parser:
 %
\begin{verbatim}
/* header file for a prettyprinting parser */

/* NB: This file is NOT protected against double inclusion! */

#include "Pparse.h" // include abstract base class

#ifndef PPARSE_NAME
#define PPARSE_NAME Ppparse
#endif

class PPARSE_NAME : public Pparse {
public:
  PPARSE_NAME();
  ~PPARSE_NAME();
  int prettyprint(istream*, ostream*);
  int prettyprint(istream*, Output*);

  void debug_on();
  void debug_off();
};
\end{verbatim}
 %
This file resides in the \t{include} directory of the \pretzel{}
distribution and together with the header of the abstract base class
is also is found in the \pretzel{} \t{include} directory of the system
(if \pretzel{} is installed).

\subsection{Example}

A minimal file that utilizes a \pretzel{} prettyprinter could look
like this:
 %
\begin{verbatim}
#include <iostream>
#include "Ppparse.h"

int main () {
    Ppparse prettyprinter;
    prettyprinter.prettyprint(&cin,&cout);
    return(0);
}
\end{verbatim}
 %
Firstly, the actual prettyprinter class is included. Then we declare a
simple prettyprinter and simply call \id{prettyprint}.  Here, the
input and output streams are connected to the standard input and the
standard output. If this \id{main} routine is compiled and linked with
the objects produced from the output of \pretzel{}, the prettyprinter
will work as specified in the two input files. The prettyprinting
parser internally calls the prettyprinting scanner by its default
interface.



\section{Building a \pretzel{} prettyprinter by Hand}

If you need to produce a prettyprinting module by hand you'll have to
directly invoke \pretzel{}, flex, Bison and your \CPP{} compiler in the
right order. To ease this, you can use the \t{Makefile} in the \PASCAL{}
subdirectory of the distribution (\t{languages/pascal}).

To produce (only) the prettyprinting parser from the file \t{pascal.fg}
for example, you could type the following:
 %
\begin{verbatim}
 $ pretzel -g pascal
This is pretzel, version 2.0.
Processing the formatted grammar file (pascal.fg -> pascal.y).
No errors found.
 $ bison -d pascal.y
pascal.y contains 21 shift/reduce conflicts.
 $ mv pascal.tab.h ptokdefs.h
 $ g++ -c -I$PRETZEL_INCLUDE -g pascal.lex.c
\end{verbatim}
 %
We assume that the \t{PRETZEL\_INCLUDE} environment variable points to
the \pretzel{} include directory.  The \t{-d} option of Bison will
produce a token header file that the scanner expects (the scanner must
know, which tokens the parser awaits and which token codes they have).
The scanner expects these definitions under the name \t{ptokdefs.h}.

\index{pretzelinclude@\t{PRETZEL\_INCLUDE} environment variable}
\index{d@\t{-d} option of Bison}
\index{token header file}
\index{ptokdefs@\t{ptokdefs.h} header file}

The above sequence of commands should leave a \t{pascal.tab.o} file in your
directory. This is the object code of the prettyprinting parser with the
default interface (i.e.~the generated class' name is \t{Ppparse} and is
declared in the include file \t{Ppparse.h}). You can now use this
prettyprinter in your own programs: by including the header file the
prettyprinter class will become visible; by linking the object with your
own program you'll get your executable. (Note that you'll need to
provide a prettyprinting scanner youself in this case or simply produce
a suitable one in a similar way using \pretzel{}.)







\section{Obtaining a \pretzel{} Prettyprinting Module}

In the preceding section I have explained how to produce a \pretzel{}
prettyprinter by hand. Now we'll see how you can change the default
interface of the produced prettyprinting scanner and parser.

\index{prettyprinting module}


\subsection{The Prettyprinting Scanner}

Say, you need a prettyprinting scanner class called \id{Fooscan}
in a header file called \t{Fooscan.h}. All you
have to do is to make a copy of the default header file \t{Ppscan.h}
(which you can find in the \pretzel{} include directory), change
its name to \t{Fooscan.h}. Then you have to change the one single
\res{\#define} statement therin, i.e.~instead of
 %
\begin{verbatim}
#ifndef PSCAN_NAME
#define PSCAN_NAME Ppscan
#endif
\end{verbatim}
 %
\index{pscanname@\t{PSCAN\_NAME} macro}
 %
you'll have to write
 %
\begin{verbatim}
#define PSCAN_NAME Fooscan
\end{verbatim}
 %
Within your formatted token file, you'll have to tie in the same definition
like this:
 %
\begin{verbatim}
%{
#define PSCAN_NAME Fooscan
%}
\end{verbatim}
 %
It is important that this text appears in the declarations section
of the formatted scanner file, i.e.~before the first line that
contains the double percent sign `\verb"%%"'.

Compiling and linking the source produced by \pretzel{} and flex
you'll get the object code that contains the desired prettyprinting
scanner. The scanner expects to find a token header file under the
default name \t{ptokdefs.h} with definitions of the symbolic names and
token codes of the tokens it should be able to scan.  Either you
produce this file by hand or you let Bison do the job for you (using
the \t{-d} option if you have a prettyprinting grammar).  Look into a
standard \t{ptokdefs.h} file to see what it should contain. The name
of the token header file \t{ptokdefs.h} may also be changed. You have
to define the \t{PTOKDEFS\_NAME} macro within your formatted token
file like this:
 %
\begin{verbatim}
%{
#define PTOKDEFS_NAME "fooscantoks.h"
%}
\end{verbatim}
 %
Now the scanner will include the file \t{fooscantoks.h} and look
for token code definitions in there. (Again, this redefinition
must be in the declarations section of the formatted token file.)



\subsection{The Prettyprinting Parser}

Changing the interface of the prettyprinting parser works in
a similar way as just explained for the prettyprinting scanner.
The differences lie in the macro names that have to be redefined.

The name of the derived prettyprinting parser class is \t{PPARSE\_NAME}.
If you say in the definitions section of the formatted grammar file
for example
 %
\index{pparsename@\t{PPARSE\_NAME} macro}
 %
\begin{verbatim}
%{
#define PPARSE_NAME Foopp
%}
\end{verbatim}
 %
then your prettyprinter class will be called \t{Foopp}. Internally
it expects the prettyprinting scanner with it's default interface
(i.e.~under the names \t{Ppscan} for the class and \t{ptokdefs.h}
for the token header file). If you redefine the same macros as in
the formatted token file, then you redefine the interface under
which the prettyprinter expects the scanner. So lines like these
in the formatted grammar would urge the prettyprinter to use
a \t{Fooscan} class object for scanning:
 %
\begin{verbatim}
%{
#define PPARSE_NAME Foopp
#define PSCAN_NAME Fooscan
%}
\end{verbatim}


\section{Multiple \pretzel{} Modules in the same Program}
\label{sec:multiple:modules}

\index{muliple modules}

Using multiple \pretzel{} modules within a single program has been
greatly simplified by introducing the object oriented interface
that has been explained in section \ref{sec:pretzel:interface}.
The main thing is to change the names of the two scanners/parsers
produced by \pretzel{} to something distinct and then to compile
and link then together. That's the theory.

But there's a problem here: the scanners and parsers produced by flex
and Bison contain a couple of global variables whose names will clash
when you try to link them together. This may be circumvented by using
special options of flex and Bison to change the prefix of these globals
when a scanner/parser is built. As I haven't needed this feature in the
past (but only tested it),  I haven't prepared an additional option to
\t{pretzel-it} to do this in a nice and easy way. At the moment you'll
still have to consult the manual pages \t{flex(1)}, \t{flexdoc(1)} and
\t{bison(1)} for details. If this is a feature you need, tell me and
I'll add it to \t{pretzel-it} (see also chapter \ref{chap:future:work},
``Future Work'').

\index{flex} \index{flexdoc} \index{Bison}

Another thing that sounds interesting, but that I have not tested yet,
is to use several instances of a single prettyprinter class produced
by \pretzel{} in the same program. If you have ever tried it, I'm
very interested in your results (see chapter \ref{chap:future:work},
``Future Work'' again).



\section{Prettyprinting for non-\LaTeX{}ians}
\label{sec:non:latex}

\begin{quote}

  ``Of course \CWEAVE{} does visual formatting, at least for the part
  that cannot be left to be visually formatted by \TeX: it issues
  forced line breaks, explicit changes of indentation level etc. The
  logical markup is really contained in the source document, and not
  (fully) in the intermediate file produced by \CWEAVE. However
  \CWEAVE{} does not do complete visual formatting: for instance many
  symbols like operators are produced as macros, so that you can
  easily change their appearance by overriding the default
  definitions, and breaking of long lines is also left to \TeX's line
  breaking algorithm.''

  Marc van Leeuwen \cite{vanLeeuwen94:use:of:prettyprinting}

\end{quote}

\index{Leeuwen@van Leeuwen, Marc}

\noindent \pretzel{} was build with \LaTeX{} in mind, but this doesn't
necessarily mean that you are restricted to using this as back end
of your prettyprinter. Through the way that \pretzel{} was designed,
it is quite easy to extend it to work with any other markup formatting
system. However, if you are opting for immediate screen output or
things like \HTML{}, you're not lost either. First, we'll see
how to handle the first case, and then let's go and look at
things like \HTML{}.

\index{HTML}
\index{latex@\LaTeX}

\subsection{Other Markup Formatters}

Internally, all pieces of code that handle the direct output of
characters or format commands to the output stream are encapsulated in a
\CPP{} class called \res{Output}. The interface and implementation of
this class are described in the \noweb{} source file \t{output.nw} in
the \t{output} subdirectory of the \pretzel{} distribution. From looking
at this file, you'll see that there is an abstract base class
\res{Output} and that the standard class, which \pretzel{}
prettyprinters use, is the \res{Latex\_cweb\_output} class\footnote{The
name of this class results from the goal, to give a \LaTeX{} \CWEB{}
touch to the output \cite{Schrod:CWEB:LaTeX}.}, a subclass of
\res{Output}. It is within this subclass that the actual translation of
the format command \fmt{force} into the string ``\verb+\pretzelforce{}+''
takes place for example.

If you are going for something like Troff, you can similarly derive a
new subclass of \res{Output} (for example by copying the original class)
and simply replace the text bits like ``\verb+\pretzelindent+'' with the
strings appropriate for Troff. Now, you have to instruct the generated
prettyprinter not to use the default (\res{Latex\_cweb}-) output, but to
use your new one. For this sake, the second version of the
\id{prettyprint} member function of the generated prettyprinter may be
used (see section \ref{sec:pretzel:interface}, ``The \pretzel{}
Interface''). Simply pass an object of your new output class to the
function and the prettyprinter will use it instead. By deriving a new
subclass, Troff and \LaTeX{} prettyprinters may coexist peacefully.

\subsection{Going for \HTML{}}

What if you don't have a real markup formatting system at hand and
want to go for something in the range of a vt100 terminal, line printer
or \HTML{} page?

Bad news is that you'll have to do without the full blown format command
set. Proper interpretation  of the line breaking commands
\fmt{break\_space} and \fmt{opt}, the commands \fmt{backup} and
\fmt{no\_indent} cannot be guaranteed in these cases. However, good
news is that the \pretzel{} distribution contains a derived
\res{Output} class called \res{Ascii\_output} that may be used
to output prettyprinted code with this restricted command set to
the screen (or to any device suitable for {\sc ASCII} characters).
The format commands that cannot be executed properly are still
valid, but result in no actions when outputting code. See the
file \t{asciioutput.nw} in the \t{output} subdirectory of the
\pretzel{} distribution for details.\footnote{An example \noweb{}
prettyprinter that emits \HTML{} code can be found in the
\t{contrib/noweb/cee.html} directory of the distribution. See also
chapter \ref{chap:pretzel:meets:noweb}, ``\pretzel{} meets \noweb{}''.}

\index{ASCII}







% ######################################################################

\chapter{\pretzel{} meets \noweb{}}
\label{chap:pretzel:meets:noweb}

\begin{quote}

  ``\noweb{} is designed to meet the needs of literate programmers
  while remaining as simple as possible. [\ldots] The primary
  sacrifice relative to \WEB{} is that code is not prettyprinted.''

  Norman Ramsey \cite{Ramsey:noweb:www}

\end{quote}

\index{Ramsey, Norman}

\noindent The \noweb{} system \cite{Ramsey:j-IEEE-SOFTWARE-11-5-97} by
Norman Ramsey is a simple tool for literate programming, a programming
style first introduced and named by Donald Knuth in
1984~\cite{Knuth:CJ-27-2-97}. The main goal of this new style is to
enhance the readability and understandability of a program's source code
to an extent that makes it enjoyable to read. This section will explain,
how the \pretzel{} system relates to this new ``paradigm''
\cite{Cordes:COMPUTER-24-6-52} and how it can be especially usefull in
conjunction with \noweb{}.

\index{Knuth, Donald E.} \index{literate programming} \index{noweb}

This chapter explains how you can use \pretzel{} prettyprinters within
\noweb{}. It expects that you know what literate programming is about
and that you have had a go at using the \noweb{} system. The articles by
Knuth \cite{Knuth:CJ-27-2-97}, Bentley \cite{Bentley:CACM-29-5-364} and
Denning \cite{Denning:CACM-30-7-593} provide good and readable
introductions to the field, while the paper by Cordes and Brown
\cite{Cordes:COMPUTER-24-6-52} discusses the paradigm and the book by
Knuth \cite{Knuth:LP92} provides a collection of related articles.
Concerning \noweb{}, there is a lot of good introductory material
available online (see for example the \noweb{} home page
\cite{Ramsey:noweb:www}). As you will need \noweb{} anyway to install
and run \pretzel{}, it is a good idea to try a small example now if you
are unaquainted with it.  Late sections will need detailed knowledge
about the concepts of noweb as explained in the original paper by Ramsey
\cite{Ramsey:j-IEEE-SOFTWARE-11-5-97} or the ultimate reference to
\noweb{}'s interior by the same author \cite{Ramsey:noweb:hacking}.
The \noweb{} extensions of \pretzel{} are still very much experimantal
and far from making \noweb{} as powerfull as language dependent
literate programming tools.



\section{Prettyprinting in \noweb{} -- How it works}

\begin{quote}

  ``They [the prettyprinting filters] can perform either or both of
  two tasks:
  \begin{itemize}
  \item choose fonts and glyphs to represent each source token
  \item choose indentation and line breaks of your code
  \end{itemize}
  I find these features of more cost than benefit (except possibly
  when preparing for book publication), but lots of people like them.

  Noweb takes the reasonable position that the programmer is the best
  judge of where to put the line breaks and how much to indent code.
  In some languages (Miranda, Haskell, awk, Icon), line breaks and/or
  indentation carry meaning, and to change them would be to change the
  meaning of the user's program.''

  Norman Ramsey \cite{Ramsey96:noweb:filters}

\end{quote}

\index{Ramsey, Norman}


% avoid UNIX terminology

\noindent A primary design goal in developing \noweb{} was to have a
literate programming tool that was simple enough to learn  easily, and
also suitable for easy modification and extensions. This second goal was
achieved by it's ``pipelined architechture'' \cite{Ramsey:noweb:hacking}.
Internally \noweb{} converts the input files into a stream of items
which --- like in a pipeline --- are squeezed through special programs
called {\em filters} that perform simple transformations on these data
items. The format of the data items in the pipeline is called
the ``noweb pipeline representation'' which interested readers can
study in detail elsewhere \cite{Ramsey:noweb:hacking}. Devising a
prettyprinter for \noweb{} means building a prettyprining filter for
it. Such a filter can then be inserted into the pipeline.

\index{noweb}

\subsection{A \noweb{} Prettyprinter for \C{}}

In the \t{contrib/noweb/cee.latex} you can find the definitions of
a simple prettyprinter for \C{}. Change into this directory and
look at the files \t{cee.ft} and \t{cee.fg}. They are normal
\pretzel{} input files that have been modified slightly to work with
\noweb{}. Before we look at these modifications, let's try to
build a \noweb{} prettyprinting filter first. To do this, you
can use the \t{-n} option of \t{pretzel-it} in the following way:
 %
\begin{verbatim}
pretzel-it -n cee prettycee
\end{verbatim}
 %
This will produce a \noweb{} prettyprinter called \t{prettycee} in
the current directory.\footnote{The \noweb{} extensions of \pretzel{}
must have been installed on your system in order for this to work.
See the top \t{README} file of the \pretzel{} distribution for details.}
This filter can then be sent into action using the \t{-filter} option of
\t{noweave}. For an example, type:
 %
\begin{verbatim}
noweave -delay -filter prettycee ceetest.nw > ceetest.tex
\end{verbatim}
 %
This will mangle the \noweb{} input file \t{ceetest.nw} and output
\LaTeX{} source to \t{ceetest.tex}. Now run it through \LaTeX{} and have
a look at the result.\footnote{People who are interested in the
internals of this operation should consult the file \t{pretty.nw} which
contains the \noweb{} prettyprinter API and the file \t{nowebpretzelpp.nw}
which contains a description of the interface between \pretzel{} and
\noweb{}. Both can be found in the directory \t{contrib/noweb/general}
of the \pretzel{} distribution.}  What do you think?

\index{n@\t{-n} option of \t{pretzel-it}}
\index{noweb!prettyprinter API}



\subsection{A \noweb{} Prettyprinter for Java}

A pretty good \pretzel{} prettyprinter that works with Java and noweb
can be found in the \t{contrib/noweb/java.latex/} directory of the
\pretzel{} distribution. It features a fine tuned prettyprinting
grammar that is quite robust (contributed in core by Lee Wittenberg),
correct typesetting of class names (in bold, as god ment them to be)
and rudimentary automatic indexing facilities.

\index{Java}
\index{Indexing with \t{noweb}}
\index{Wittenberg, Lee}

After installing \pretzel{} and making the file \t{prettyjava} (by
simply typing `\t{make}'), try the filter on Java files of your own.
The command to use then is:
 %
\begin{verbatim}
noweave -delay -filter prettyjava -index file.nw > file.tex
\end{verbatim}
 %
\index{delay@\t{-delay} switch}
\index{filter@\t{-filter} switch}
\index{index@\t{-index} switch}
 %
Note that the \t{-filter} switch comes before \t{-index}. To get this
stuff typeset correctly, your file needs to access the
\t{pretzel-noweb.sty} \LaTeX\ style file (which gets installed when
\pretzel{} gets installed). For this reason, you need to input it
withing your document. The frames of my \noweb{} files always look
like this:
 %
\begin{verbatim}
\documentclass{article}
\usepackage{noweb}
\usepackage{pretzel-noweb}
%
\begin{document}

... % rest of the noweb file

\end{document}
\end{verbatim}
 %
Note that you still have to include the \t{noweb.sty} package before
accessing \t{pretzel-noweb.sty}.

\index{pretzelnoweb@\t{pretzel-noweb.sty}}
\index{nowebsty@\t{noweb.sty}}

This prettyprinter shows, how it can be made possible to include
explicit format commands within the code a la \CWEB{}. For example
you can say
 %
\begin{verbatim}
a:=1; //@cancel
\end{verbatim}
 %
and the \fmt{force}d line break that follows every statement will be
canceled at this place. The only other format command that is possible
here is \fmt{force}, which can be coded as \verb"//@/" (a little like
\CWEB{}) or \verb"//@force", but in practice all other format commands
can easily be added. Note that the format commands are comments in
Java so they don't bother the tangling process.

\index{cweb@\CWEB{}}
\index{format commands!extra}
\index{force@\fmt{force}}
\index{cancel@\fmt{cancel}}

\subsection{Writing Prettyprinting Grammars for \noweb{}}

\begin{quote}

  "The most significant downsinde of not using \CWEB{} is lack of code
  prettyprinting; however, after prettyprinting code for quite some
  time I got tired of it and I am now a nuweb minimalist."

  Przemek Klosowski \cite{klosowski96:cweb:pp}

\end{quote}

\index{Klosowski, Przemek}


\noindent Now, how do the \t{cee.ft} and \t{cee.fg} files differ from
their non-\noweb{} counterparts? From your point of view, consider every
code chunk as a seperate input to the prettyprinter you write. So the
only things to note when building a prettyprinter for \noweb{} is that
you may have arbitrary code fragments instead of full blown programs or
functions, and that chunk uses may appear anywhere within the input code.

Chunk uses come as individual lines into the prettyprinter. These
lines are in raw pipeline representation, i.e.~look something like
this:
 %
\begin{verbatim}
@use Foo Bar
\end{verbatim}
 %
The scanner must detect these lines and basicly must wrap them up
into a token that doesn't get changed. Thus, the formatted token file
contains a line like this:
 %
\begin{verbatim}
^"@use\ ".*          CHUNK  { "\n" ** "\n" }
\end{verbatim}
 %
The `\verb"^"' ensures that the input matches from the beginning of
the line. The two newlines in the attribute definition will result in
the line being passed out of the prettyprinter unchanged. (The
prettyprinter internally accumulates the code lines of an entire code
chunk and then prettyprints it into a string. Then he cuts the
prettyprinted code into lines before inserting it into the pipeline
again. Lines that don't start with an `\verb"@"' are prefixed with
``\verb*"@text "''.) This chunk token now can be used within the
formatted grammar.

Because such a chunk token may appear everywhere in the text, it
would be nice to have some rule that will stick a chunk token to
any token that precedes it. As we have no possibility of using
wildcards in token names, we use a simple trick: at the end of
the grammar there is a rule like this for every possible token
name:
 %
\begin{verbatim}
^"@use\ ".*     CHUNK   { "\n" ** "\n" }
^"@".*          IGNORE  { "\n" ** "\n" }
\end{verbatim}
 %
The special \t{IGNORE} tokens result from another line in the
formatted token file. This line reads as follows and is placed
behind the line that matches chunk uses:
 %
\begin{verbatim}
^"@".*          IGNORE  { "\n" ** "\n" }
\end{verbatim}
 %
As the prettyprinter input may contain other lines of internal
\noweb{} representation code (they all start with `\verb"@"'), they
need to be passed through the prettyprinter untouched too. This is
what these two rules are good for.

Note that the \t{ceetest.nw} file includes the \t{noweb.sty}
\LaTeX{} style file as well as the file \t{pretzel-noweb.sty}.
It is important that the \t{noweb} style comes before the
\t{pretzel-noweb} style, because the latter redefines a macro
from the prior (see also section \ref{sec:noweb:problems}).


\subsection{Debugging}


You may debug the prettyprinting filter by setting an environment
variable called \t{PRETZEL\_NOWEB\_DEBUG} to the value ``\t{on}''.
Debug information will be turned off again when you \t{unset} the
variable. This is a simple, but easy to use method of controlling
debug output. The output is much like the one explained in section
\ref{sec:debugging} on page \pageref{sec:debugging}, and the tips and
tricks from section \ref{sec:tips:and:tricks} come in handy too.
Suggestions for improvements are welcome.

I regularly run the parsing within Emacs (using the shell command
possibility) ans then can easily search the debug output and follow
how the tokens are put together.




\subsection{Making the Best Use of It}

\begin{quote}

  ``The one area where language dependent LP tools are ahead is
  identifier indexing, but I suspect that for \CPP{}, this is an almost
  impossible task anyway.''

  Matthias Neeracher \cite{neeracher96:lp:indexing}

\end{quote}

\index{Neeracher, Matthias}

\noindent Having understood how \noweb{} prettyprinters differ from
normal \pretzel{} prettyprinters, it is easy to convert files in either
direction. But now, we also get language dependent information within
the prettyprinter nearly for free and we can for example start to build
an index of identifier definitions and uses.

For example, in the formatted token file, you could write:
 %
\begin{verbatim}
[a-zA-Z][a-zA-Z0-9_]*    ID
                         {"%\n@index use " + ** + "\n"
                          "{\\it "  [escaped_underlines(yytext)] "}" }
\end{verbatim}
 %
Here, every identifier will finally be preceded by a line that will
tell the indexer of \noweb{} about the appearance of the identifier in
this chunk. See the ``\noweb{} Hacker's Guide''
\cite{Ramsey:noweb:hacking} for details on the keywords that are
allowed. But note, that if you're doing it this way the \t{-index}
option of \t{noweave} has to appear {\em after} the \t{-filter} option
on the command line.


\subsection{Some Naming Conventions}

\index{naming conventions}

If you are looking at example files of \noweb{} prettyprinting filters,
it's quite handy to notice some conventions that I have followed.
The formatted grammar and the formatted token file should carry a name
as a clear indication what programming language they contain (e.g.
\t{cee.ft} or \t{java.fg}). The prettyprinting filter for \noweb{}
that is produced should be called \t{pretty...} with the dots replaced
by the programming language name (e.g.~\t{prettycee} or
\t{prettyjava}).

In the development directories you'll find special \noweb{} files
meant to test the prettyprinters, not to be of any programming use.
They include most of the syntax constructs of the programming language
in question. These files are very instructive when building a new
prettyprinter for noweb or testing one that you have changed.







\section{Problems}
\label{sec:noweb:problems}

\index{noweb!problems}

\begin{quote}

  ``I think the problem here is not so much that prettyprinting is
  inherently bad as that many languages don't benefit from it so much.
  \ALGOL{} and \PASCAL{}, I believe, benefit quite a bit from
  appropriate prettyprinting.  These are languages where you write out
  a lot of whole words: `begin', `end', `do', `then' \ldots\ You get
  the idea.  Languages like \C{} and Icon depend a lot on symbolic
  notation.  For the large part, all \CWEB{} does is replace one set
  of symbols with another that some people happen to prefer.  But,
  still, \CWEB{} puts reserved words and defining words in boldface
  and identifiers in italics; what of that?  In \PASCAL{}, doing that
  helps mark out the form of a construct.  In \C{}, on the other hand,
  the constructs are delimited by things like parentheses and braces,
  and so putting things in different typefaces doesn't help that much.
  OK, \CWEB{} indentifies, but you don't need weave to do that; Emacs
  can do that for you, or you can do it yourself, and the web source
  will benefit from it.  The main thing that I might want from
  \noweb{} but don't get is typesetting of comments, and I can live
  without that because I don't need a great many comments.''

  Barry Schwartz \cite{schwartz94:use:of:prettyprinting}

\end{quote}

\index{Schwartz, Barry}

\noindent Handling of comments in prettyprinting remains an
unsolved problem to date (see section \ref{sec:open:pp:problems}).
This is partially due to the fact that that comments can crop up
anywhere in the code and that they themself can again contain
quoted code. With \noweb{} this problem isn't so severe, because
chunks mostly do not contain a lot of comments as they are moved
to the documentation parts of the literate program. But still
a nice way of handling comments needs to be found.

Also, quoted code in documentation chunks isn't prettyprinted. This
is partially a deficit of the prettyprinter API and partially a
structural problem. One would need two distinct prettyprinters
to handle normal chunk code and quoted code seperately because
you don't want to have forced line breaks within documentation.

Another problem arises from the \t{noweb.sty} \LaTeX{} style. The
environment for setting code naturally is rather restrictive and a macro
needs to be changed in order to allow the \pretzel{} macros to work
correctly.\footnote{I'm not a \TeX{} freak after all. Thanks to Lee
Wittenberg for his help on this.} All this signals, that  \pretzel{} an
\noweb{} have not become real friends yet; they still have to work on
their relationship.

\index{noweb.sty}


% ######################################################################

\chapter{On Prettyprinting}
\label{chap:on:prettyprinting}

\begin{quote}

  ``\noweb{} does automatic indexing and cross-referencing for some
  languages, including \C{}, Icon, \TeX, and Standard ML.  Some people
  have made it prettyprint other language, like Icon and
  Object-Oriented Turing.  I haven't been overwhelmed by the results,
  but then I'm notoriously difficult to convince of the value of
  prettyprinting.''

  Norman Ramsey \cite{ramsey94:use:of:prettyprinting}

\end{quote}

\index{Ramsey, Norman}


\noindent The term {\em prettyprinting} means the rearrangement of a
program's source code to illuminate it's logical structure and thus
enhance readability. The term goes back to Henry Ledgard and his
``Programming Proverbs''~\cite{Ledgard75}. Prettyprinting has become a
field of interest because many of todays popular programming languages
are so called ``free-format'' languages, where there are basically no
column-position or line-boundary restrictions on statements,
declarations, or comments.

\index{Prettyprinting}\index{free format@``free format'' languages}
\index{Ledgard, Henry F.}

As a matter of fact, formatting and all other aspects of readability
depend heavily on personal taste and skill.  This has lead to a wide
range of suggestions and standards concerning the formatting of
programming languages, especially for the \PASCAL~\cite{Jensen85}
language~\cite{Arab92,Bailes84,Bates81,Bond79,Crider78,Grogono79,Gustafson79,Heckert85,Ledgard77,Marca81,Peterson77,Mohilner78}.\label{formattingstandards}
The consistency of these proposals has also enabled construction of
automatic formatting algorithms and tools that are usually called {\em
  prettyprinters\/} (or {\em indenting programs\/}).  In the
literature general examples of prettyprinters have been presented
among others for the languages \ALGOL~\cite{McKeeman65,Scowen71},
PL/I \cite{Conrow70},
\LISP~\cite{Goldstein73,Hearn79,Waters83,Waters92}, Ada
\cite{Norris84}, and of course for
\PASCAL~\cite{Hueras77,Jackel80,Bates81,Winter89,Arab92}.

\index{Personal taste}\index{Taste!personal}\index{pascal@\PASCAL}
\index{Formatting standards}\index{Formatting algorithms}
\index{Prettyprinters!for other languages} \index{Indenting
  programs|see{Prettyprinters}}
\index{algol@\ALGOL}\index{PL/I}\index{lisp@\LISP}\index{Ada}
\index{pascal@\PASCAL}

However, technological advance in the area of automatic typesetting
has led to a set of powerful formatting tools called {\em
typesetters}, or {\em typesetting systems}, of which
\TeX~\cite{TeXbook} is maybe the most apparent to computer scientists.
These tools allow the preparation of documents in high quality suited
for publication. Among the features of these systems are automatic
line- and page-breaking, powerful macro-processing facilities and
convenient devices for typesetting mathematical formulas. Seen in this
light, automatic program formatting is only a simple instance of
automatic typesetting \cite[p.~652]{Rose81}. So prettyprinting
algorithms that rely on professional typesetters can be simpler and
produce better results at the same time, because a lot of the
formatting problems (e.g.~alignment, line-breaking, typesetting
mathematical formulas) can be taken on by the typesetter.

\index{Automatic typesetting}\index{Typesetters}\index{Typesetting
  systems}\index{tex@\TeX}\index{Program formatting}

In this chapter deals with the general issue of prettyprinting. It
places a wider view on the field and tries to locate the \pretzel{}
system within the latest research.



\section{Prettyprinting with Format Commands}
\label{fmt-commands}

\begin{quote}

  ``Well, I must confess, I was very close to throwing the whole
  \CWEB{} away and switching to something without any prettyprinting,
  like {\tt nuweb} or \noweb{}. I installed {\tt nuweb}, rewrote the
  {\tt Matrix2D} example into it, ran it through and -- returned back
  to \CWEB{}. I probably got spoiled by the nice-looking output it
  produces -- almost always.''

  Jan Dvorak \cite{dvorak95:cweb:nice:output}

\end{quote}

\index{Dvorak, Jan}

\noindent The prettyprinting method used by \pretzel{} can be called
{\em prettyprinting with format command primitives} and goes back to a
similar method used by Knuth in his original \WEAVE\ system for \PASCAL\
(see section \ref{relatedwork} for details). Like most other
prettyprinting algorithms this method functions similar to a {\em
compiler}, i.e.~an input file is processed and translated into an output
file according to special rules. Here, the input file is the patch of
source code that shall be prettyprinted and the output file is the text
suited for the typesetter. The translation rules represent the special
way in which source code should be formatted.

\index{pascal@\PASCAL}\index{weave@\WEAVE}\index{Knuth, Donald E.}
\index{Compilers!and prettyprinters}

The main assumption is that we have a text formatter that uses
{\em control sequences\/} (or {\em tags\/}) inside the actual text body
for specifying all the different ways of formatting (such as
fonts, indentation, spacing, etc.). Another term for this way of
formatting control is {\em in-text procedural markup\/} and systems
using this technique are commonly called {\em document compilers}
(though this might only refer to the way they work, not to viewing
text processing as programming). Common examples of such typesetting
systems are \TeX\ and the \UNIX\ tools Nroff/Troff.
 %
\index{Control sequences}\index{Tags}\index{In-text procedural markup}
\index{Markup}\index{Procedural markup}\index{Document compilers}
\index{Compilers!for documents}\index{tex@\TeX}\index{unix@\UNIX}
\index{Nroff}\index{Troff}
 %
During the processing of the input the only thing it does is to enrich
the stream of incoming information with special control sequences that
I call {\em format commands} (or {\em format command
primitives}). These commands are tags that are left for the typesetter
to interpret. The set of format commands used by \pretzel{} is basically the
initial set used by Knuth.
 %
\index{Knuth, Donald E.!command set by}
 %
To understand this procedure, see chapter \ref{chap:using:pretzel}.
The original documents by Knuth \cite{Knuth:web83} and Knuth and Levy
\cite{Knuth:CSS93} present the algorithm in (in)formal detail.



\section{A Short History of Prettyprinting}
\label{relatedwork}

\begin{quote}

  ``Nevertheless I strongly disagree with people saying that
  pretty-printing is not worth the effort for anyone in general; this
  depends very much on ones particular situation and purpose. In some
  cases pretty-printing may even be the main reason to opt for
  literate programming.''

  Marc van Leeuwen \cite{vanLeeuwen95}

\end{quote}

\index{Leeuwen@van Leeuwen, Marc}

\noindent In this section I will give a brief chronological overview
over the area of prettyprinting, try to structure the field, and will
show, how the present system fits into it.

\index{History of prettyprinting}\index{Prettyprinting!history}



\subsection{Historical Notes}

The prettyprinting of code has a long tradition that not only
originates from the compulsion to typeset programs for publication.
The readability or even the sheer beauty of a prettyprinted algorithm
have also been a key motivation to this part of computer science.

\index{Tradition}\index{Beauty}

The first people to call special attention to formatting issues were
probably Peter Naur, Myrtle Kellington and William McKeeman. While
Myrtle Kellington, as executive editor of ACM publications, helped to
develop high quality programming-language typography standards, Naur
was the first to include such formatting standards into his report on
the \ALGOL~60 language \cite{ALGOL-Report} and McKeeman was the first
to present a prettyprinting algorithm for it \cite{McKeeman65}. This reaches
back to the early 60s. In fact, McKeeman`s algorithm is the first
actual prettyprinter to be found in the literature. Its purpose was
 %
\index{Naur, Peter}\index{Kellington, Myrtle}\index{McKeeman, William}
\index{ACM}\index{algol@\ALGOL}\index{Prettyprinter!first}
 %
\begin{quote}

``[\dots] to edit \ALGOL~60 text that is difficult to read
because, for example, the \ALGOL\ has been transcribed from
printed documents, or written by inexperienced programmers [\dots]''

\end{quote}
 %
Later, other prettyprinters were presented for PL/I~\cite{Conrow70},
\LISP~\cite{Goldstein73}, and finally \PASCAL~\cite{Hueras77}.
These systems simply looked at the input text searching for special
keywords. When encountering a keyword (which could be a \res{begin} or
an opening brace, for example) a certain action would be triggered
which would result in a line break, change of indentation, etc.

\index{PL/I}\index{lisp@\LISP}\index{pascal@\PASCAL}
\index{Prettyprinters!running on keywords}

In these first approaches  to the issue of prettyprinting the basic
actions that prettyprinting includes were already visible. These
concern (in order of importance):\footnote{In the late 70s there was
  one additional point on this list, namely the introduction of `connector
  lines' into the prettyprinted output \cite{Clifton78,Ramsdell79}. But
  this method hasn't found too much support since.}

\index{Connector lines}\index{Prettyprinters!basic actions}
\index{Basic actions of prettyprinters}

\label{basic-actions}

\begin{itemize}

\item indentation and folding (i.e.~determining line breaks) of source code

\item the additional spacing of syntactic constructs (like expressions)

\item using additional typographic means (e.g.~different fonts)

\end{itemize}

\index{Indentation}\index{Folding}\index{Spacing!additional}
\index{Additional spacing}

The main disadvantage of the early systems was their language
dependence.  The style of prettyprinting was hard-wired into the
system, which was basicly a big ``case'' switch over all known
constructs that were to be treated specially. The main advantage of
them, however, was their simplicity and their error handling
capabilities (which is also the reason why such and similar tools are
still widely used today). But it was soon clear that this approach
wasn't adequate and that structural changes had to be made to the
concept.

\index{case@``case'' construct}\index{Early prettyprinters}
\index{Prettyprinters!early systems}

In their paper titled ``A One-pass Prettyprinter'' \cite{Hearn79}
Hearn and Norman introduce a fundamentally new idea into this area.
They simplify the whole concept of prettyprinting by introducing
structure.

\index{Hearn, A. C.}\index{Norman, A. C.}

\begin{quote}

  ``The new method that we propose here will be described in terms
  of a pair of coroutines. One of these will be responsible for
  producing a stream of characters that represent the program
  being printed, the other makes decisions about how these characters
  should be displayed.'' \cite[p.~52]{Hearn79}

\end{quote}

This is a fundamental distinction, namely that between {\em formatting
  policy\/} and {\em formatting algorithm\/} \cite{Bailes84} and is a
vital step on the road to language independence.\footnote{This idea
  however has also introduced a slight blur in the
  terminology, as some authors refer to prettyprinting as consisting
  only of the actual ``printing'' on paper (i.e.~the formatting
  algorithm) \cite{Oppen80,Jokinen89} and others still mean the entire
  process.}

\index{Formatting policy}\index{Formatting algorithm}
\index{Algorithm-policy distinction}\index{Policy-algorithm distinction}
\index{Coroutines}

A problem that arises from this separation is: how do the two coroutines
communicate? Hearn and Norman use a simple FIFO buffer in which the first
coroutine inserts text and ``special markers'' that indicate its
decisions on the policy. These markers were mainly special blanks that
indicate possible line breaks.

\index{FIFO buffer}\index{special markers@``special markers''}

\begin{quote}

  ``The markers will contain enough information for the formatting
  process to discover what level of indentation would be appropriate
  to use were a line break to be inserted at that point.''

\end{quote}

This resembles a special kind of `communication protocol' and is the
first hint to a thing that I have called `format command primitive'
throughout this chapter. Other authors
\cite{Jackel80,Oppen80,Waters83,Blaschek89,Rubin83,Rose81,Jokinen89}
have taken up this idea and have introduced different sets of format
commands that change and increase the communication facilities between
these two processes.  The two most elaborate to date are those of
Rubin \cite{Rubin83} and that of Knuth and Levy \cite{Knuth:CSS93},
that is used here.

\index{Communication protocol}\index{Format commands}
\index{Rubin, Lisa F.}\index{Knuth, Donald E.}\index{Levy, Silvio}

The separation introduced by Hearn and Norman is an example of `separation
of concerns': the language dependent parts of prettyprinting are
separated from the actual typesetting issues that arise, when putting
text to paper. Blaschek and Sametinger \cite{Blaschek89} call this
a distinction between ``language dependent front end and language independent
back end''.

\index{Hearn, A. C.}\index{Norman, A. C.}\index{Separation of concerns}
\index{Language (in)dependence}\index{Prettyprinting!language (in)dependence}
\index{Blaschek, G.}\index{Sametinger, J.}
\index{Front end}\index{Back end}\index{Language dependent front end}
\index{Language independent back end}





\subsection{The Language Dependent Front End}

The earliest suggestions how to build the language dependent front end
go back to Oppens fundamental paper on prettyprinting \cite{Oppen80}. Though
his main concern is the language independent back end, he makes suggestions
about the ``preprocessor'' that should drive it. The main idea is
to make a full syntactic parse of the prettyprinted code and to use
the resulting parse tree to drive the prettyprinter. The format commands
are either assumed to be implicit in the tree (i.e.~every single branch
is treated as a structural block) or they are explicitly inserted into the
tree during the parse. Oppen writes:
 %
\index{Oppen, Derek C.}\index{preprocessor@``preprocessor'' for prettyprinting}
\index{Parse tree}\index{Implicit format commands}\index{Explicit format
  commands}\index{Format commands!explicit}\index{Format commands!implicit}
 %
\begin{quote}

  ``First, notice that the information needed by the prettyprinter
  can often conveniently be represented directly in the grammar [\dots].
  We modify the grammar of the language to contain prettyprinting information
  as above, where [\dots] [the formatting commands] are nonterminals
  mapping only the empty string.'' \cite[p.~475]{Oppen80}

\end{quote}

This aspect has at last enabled people to use formal methods to
describe layout rules. Mateti's paper \cite{Mateti83a} is another
approach in the same direction and is the first to adapt this method
to \PASCAL.\footnote{ It is interesting that Mateti's solution was the
  result of formalization (due to verification needs) and that Oppen's
  proposals emerged from the language independent nature of his
  approach.} The fact that rules for layout and grammar are often
closely related \cite{Leinbaught80} also is a strong indication that
this method is in fact adequate. The trend towards formal
specification in this area has indeed been ``very fortunate''
\cite{Woodman86} and has also proved to be useful in other areas, such
as syntax-directed editors \cite{Rubin83}.

\index{Formal methods}\index{Mateti, P.}\index{pascal@\PASCAL}
\index{Oppen, Derek C.}
\index{Syntax-directed editors}\index{Editors!syntax directed}

To this end Rose and Welsh \cite{Rose81} have advocated the integration
of format rules within the language syntax on the language design level.
They state that ``program format decisions [should be put] in the domain
of the language's designer, rather than its several implementators or
numerous users, which implies uniformly formatted programs of improved
readability and therefore usability.'' \cite[p.~651]{Rose81} Their idea
does {\em not} try to impose rigid formatting rules on users of
existing or future languages. Instead they propose a metasyntax and a
set of guidelines that constrain and direct the language designer in
the placing of format commands. These format commands include commands for
indentation, as well as for line breaks and optional line breaks (so
called ``fold options''). The metasyntax and the guidelines only effect
rather global issues of formatting but ensure consistency across language
borders. They also present a folding algorithm that implements the format
commands from their list.

\index{Rose, G. A.}\index{Welsh, J.}\index{Rigid formatting rules}
\index{Metasyntax (Rose and Welsh)}\index{Guidelines (Rose and Welsh)}
\index{Folding algorithm (Rose and Welsh)}

This last approach by Rose and Welsh is currently the `state of the
art' in program layout issues. Woodman has published a thorough
discussion of this scheme and has proposed refinements
\cite{Woodman86} that however do not alter the main points. It is an
approach that in some way surpasses the area of prettyprinting in that
it forces the formatting grammar to be equal to the language reference
grammar. However, Knuth shows that a prettprinting grammar can be much
simpler than a full-size language grammar, because it is able to treat
semantically different contructs equally as they are formatted in the
same way (e.g.~\res{if}, \res{for} and \res{while} statements).

\index{State of the art}\index{Woodman, M.}\index{Knuth, Donald E.}



\subsection{The Language Independent Back End}

The language independent back end implements the formatting algorithm.
It actually makes the formatting decisions that the formatting policy
has outlined and communicated to it via the format commands. The main
concern of most algorithms is line breaking (or {\em folding\/}). The
proposals by Rose and Welsh \cite{Rose81} and Oppen \cite{Oppen80} have
been the most influential ones on other systems
\cite{Waters83,Jokinen89,Blaschek89} in the literature. However, in
both articles the authors refer to the future of their proposals for
the formatting algorithm:
 %
\index{Formatting policy}\index{Folding}\index{Language independent back end}
\index{Rose, G. A.}\index{Welsh, J.}\index{tex@\TeX}
 %
\begin{quote}

  ``The changes advocated must be seen in the light of current technological
  advances. Automatic program formatting is a very simple instance of automatic typesetting, as that provided by Knuth's TEX system for technical or
  mathematical text.'' \cite[p.~652]{Rose81}

  ``It [the algorithm] is not, however, as sophisticated as it might be,
  and certainly cannot compete with typesetting systems (such as TEX) for
  preparing text for publication.'' \cite[p.~466]{Oppen80}

\end{quote}

As the \WEB\ system shows, \TeX\ has been used as a back
end for a prettyprinter, but it has remained the only one that I am aware
of. This leads to the question, why so little prettyprinters faciliate
typesetters as back ends for their output? This question is very striking,
as typesetters offer powerful capabilities, and for example, \TeX's line
breaking algorithm has been advocated to be very suitable for exactly
this task \cite{Knuth-Plass81}.

\index{web@\WEB}\index{tex@\TeX}

Oppen gives an answer to this in his paper:
 %
\begin{quote}

  ``However, it [the algorithm] seems to strike a reasonable balance
  between sophistication and simplicity, and to be appropriate as a
  subcomponent of editors and the like.'' \cite[p.~466]{Oppen80}

\end{quote}

\noindent It is surely the question whether you are prettyprinting on
screen or preparing a publication.  But the other part of the answer is
maybe not so obvious and has to do with the traditional fears in the
programming community.

A lot of prettyprinters have been built as parts of larger
programming environments and were constituent parts of this larger
system. A lack of modularity in the design (as for example the
fundamental distinction presented in these sections and visialized
in figure \ref{pp-concept}) often
makes the resuing of subcomponents impossible. The prettyprinters
hide somewhere in the larger system and are difficult to extract.

\index{Reuseability}\index{Modularity}

Another point is that an old habit of programmers is their strive
for independence. They do not want their systems to rely on other
systems. But this habit is hopefully getting less frequent today.


\begin{figure}[htb]
  \begin{center}
    \leavevmode
    \input{../figures/pp-concept.latex}
  \end{center}
  \caption{The module structure of modern prettyprinters}
  \label{pp-concept}
\end{figure}



\subsection{The Set of Format Commands}

The format commands passed between the front end and the back end of
the prettyprinter are the only means to convey the formatting policy
to the actual formatter (see figure \ref{pp-concept} that visualizes
the modularity of a prettyprinter). This means that every typesetting
feature must be expressible in them. In conjunction with this work it
is important to ask, if the command set provided in the current
\pretzel\ system is sufficient to format every desired feature that we
can possibly think of. The answer is clearly `No!', as there still are
open prettyprinting problems. We will have a look at them later. But
just how sufficient is the command set?

\index{no@``No''}\index{Sufficiency of command set}
\index{Command set!sufficiency}

First we'll look at whether it allows full control over line breaks
and indentation. A detailed comparison shows that the command set used
by Rose and Welsh is a subset of Knuth's command set (\fmt{indent},
\fmt{outdent}, \fmt{force}, \fmt{breakspace}, \fmt{opt}).\footnote{
  The commands used by Rose and Welsh are denoted by the special
  symbols {\em m, o, r, i} and {\em s}.  A stack is used to keep track
  of the indentation of the preceding lines. The symbol $m$ pushes the
  current horizontal print position onto the stack and the symbol $o$
  pops the stack again. The metasyntax insists that every string
  produced from a non-terminal is bracketed by an $m$\dots$o$ pair. As
  no $m$ and $o$ are allowed elsewhere in a rule body, the amount of
  indentation is always restored to the prevoius value during
  formatting.

  The symbol $r$ denotes a line feed and a carriage return to the
  margin that is stored in top of the stack, whereas $i$ increments
  the top-of-stack margin value and then performes an $r$. The $s$
  symbol is used to emphasize strings produced from a rule of the
  grammar. It is only allowed at the beginning of a rule body and
  directs the folding algorithm to force line breaks before and after
  the string produced from this rule. Optional line breaks (so called
  {\em folding options\/}, denoted `[$i$]' or `[$r$]') can be inserted
  between the non-terminals in the rule body.

  The most obvious analogy is the $i$ symbol. It corresponds to the
  sequence \fmt{indent}, \fmt{force} and since every indent is
  followed by a \fmt{force} or made superfluous by a matching
  \fmt{outdent} you may transform every occurence of \fmt{indent} into
  an optional $i$.  The $r$ symbol, performed inside the $i$, denotes
  a \fmt{force} command. By popping the margin stack, the $o$ symbol
  plays the role of the \fmt{outdent} command. The two fold options
  `[$r$]' and `[$i$]' correspond to \fmt{break\_space} and \fmt{opt}.}
Rose and Welsh in fact show by example, that any language syntax can
be transformed into a formatted syntax that in itself is capable of
specifying all desirable folds and changes of indentation. The main
precondition for this transformation is, that the starting grammar is
context free.  None of the five basic commands is superfluous (as we
have seen in the first part of this chapter) so we can call these
basic commands the {\em necessary command set\/}.

\index{Knuth, Donald E.}\index{Welsh, J.}\index{Rose, G. A.}
\index{Context free grammar}\index{Necessary command set}

The necessary command set enables the first point of the basic actions
of a prettyprinter mentioned on page \pageref{basic-actions}. The
other two points (additional local spacing of syntactic constructs;
using different fonts) are things that depend on the capabilities of
the typesetter and are not so much a question of the format command
set.  Thanks to procedural markup, as long as the prettyprinter allows
us to insert strings between any two tokens at lexical level we are
able to handle these two points too.  However, in Rubin's rather
assembler like command set \cite[p.~122]{Rubin83} we see that the
necessary command set still lacks full control over vertical and
horizontal spacing. One can argue over what parts of this issue belong
into the domain of the typesetter and what should be left to the
user's control, but there surely is an obvious need to express
additional vertical separation in terms of a simple format command. In
our case this is the \fmt{big\_force} command.

\index{Procedural markup}\index{Markup}\index{Lexical level}
\index{Rubin, Lisa F.}\index{Assembler like command set}
\index{Command set!assembler like}\index{Spacing!vertical}
\index{Spacing!horzontal}
\index{Format commands!big force@\fmt{big\_force}}
\index{big force@\fmt{big\_force}}

The other command that gives better control over horizontal space is
\fmt{backup}. Both \fmt{backup} and \fmt{big\_force} can be simulated
by sequences of commands from the necessary command set.\footnote{The
  sequence `\fmt{force}, \fmt{backup}' is equal to `\fmt{outdent},
  \fmt{force}, \fmt{indent}' and `\fmt{big\_force}' is something like
  `\fmt{force}, \fmt{null}, \fmt{force}', depending on the exact
  interpretation of the typesetter.} So they are not necessary but
convenient short-hands. We will call them, together with the basic
commands the {\em convenient command set\/}.

\index{Format commands!backup@\fmt{backup}}\index{backup@\fmt{backup}}
\index{Convenient command set}\index{Command set!convenient}

Typesetting lines of code flushleft is not a feature commonly needed
in modern structured programming languages, except in \C/\CPP\ when
using preprocessor directives (and still they do not belong to the
actual language itself).  The \fmt{no\_indent} command caters for them
but it is hardly necessary in other languages, where alone the idea of
excluding single lines from the overall indentation frame violates all
rules of good program layout.

\index{c@\C}\index{c++@\CPP}\index{Format commands!no indent@\fmt{no\_indent}}
\index{no indent@\fmt{no\_indent}}

Finally the \fmt{cancel} command does not add any striking new
features to the command set. It just allows more flexibility in the
placement of format commands in the output.  A well placed
\fmt{cancel} command can obliterate a dozen rules of your
prettyprinting grammar and thus can deflate prettyprinting grammars
very much. So if you define sufficiency as incorporating convenience
and flexibility we could call this whole set of presented commands a
{\em sufficient command set}. This, however, naturally excludes the
open problems mentioned in the next subsection. The two subsets of
this sufficient command set are summarized in table \ref{command-set}.

\index{Prettyprinting grammar!deflating}\index{Deflating prettyprinting
  grammars}\index{Format commands!cancel@\fmt{cancel}}\index{Sufficient
  command set}\index{Command set!sufficient}\index{cancel@\fmt{cancel}}


\begin{figure}
\begin{tabular}{rl}

necessary command set $N\quad =$ & \{ \fmt{indent}, \fmt{outdent}, \fmt{force},
        \fmt{break\_space}, \fmt{opt} \} \\
convenient command set $C\quad =$ & $N$ $\cup$ \{ \fmt{backup}, \fmt{big\_force} \} \\
sufficient command set $S\quad =$ & $C$ $\cup$ \{ \fmt{cancel}, \fmt{no\_indent} \} \\

\end{tabular}
\caption{Naming of subsets of the command set.}
\label{command-set}
\end{figure}


\subsection{Open Prettyprinting Problems}
\label{sec:open:pp:problems}

\begin{quote}

  ``No there isn't, and there won't be [a method of vertically aligning
  statements in \CWEB{}]. It is true that\CWEB{} forgets
  your alignment, but you should realise that it also forgets {\em
    any} layout feature present in your source file (e.g., line
  breaks). Apart from the fact that it would be very hard to integrate
  any alignment processing into \CWEAVE's ordinary parsing operations,
  it is an impossible task for \CWEAVE, since it knows nothing about
  the actual width of identifiers and other items. So the only
  possibility would be to transmit alignment instructions to \TeX, but
  \TeX{} typesets code in paragraph mode (breaking lines to the width
  of the page if necessary), and there is no way to mix that with an
  alignment.''

  Marc van Leeuwen \cite{vanLeeuwen96:pp:control}

\end{quote}

\index{Leeuwen@van Leeuwen, Marc}

\noindent If you have ever tried to use a prettyprinter for your own
code you will surely have come across problems concerning user control.
Yehudai states, that
 %
\index{Prettyprinting!problems}\index{User control}
\index{Yehudai, A.}
 %
\begin{quote}

  ``[\dots] it is not clear that any automated indentation scheme will
  be adequate, as I may choose different ways to lay out the same
  construct in different parts of my program.''
  \cite[p.~85]{Yehudai80}

\end{quote}
 %
Woodman gives an example for this point that he calls ``adaptive
combs'' \cite[p.~616]{Woodman86}, where a long \res{if}/\res{elsif}
statement in Modula-2 is cited as:
 %
\index{Woodman, M.}\index{adaptive combs@``adaptive combs''}
\index{Modula-2}
 %
\begin{outp}
  \res{if} \id{longexpression1} \res{then}\myindent\myforce
    \id{longstatement1}\myoutdent\myforce
  \res{elsif} \id{expr2} \res{then} \id{stmt2}\myforce
  \res{elsif} \id{longexpression3} \res{then}\myindent\myforce
    \id{longstatement3}\myoutdent\myforce
  \res{else}\myindent\myforce
    \id{longstatement4}\myoutdent\myforce
  \res{end}\myforce
\end{outp}
 %
Here the statement \id{stmt2} appears on the ``tooth'' of the comb,
whereas the context rather suggests that it should appear between
the teeth like all other statements. This is a question of {\em context
  sensitive formatting\/} and it is an open question, how to deal
with this problem in terms of format commands.\footnote{
  Woodman proposes the idea of ``linked folds'', i.e.~a special
  set of format commands that force a number of consecutive folds
  to be treated equally, but doesn't elaborate on it.}

\index{Tooth}\index{Context sensitive formatting}
\index{Linked folds}

Another open problem falls into the same domain, but deals with horizontal
spacing instead of line breaking.  Blaschek \cite[p.~701]{Blaschek89} points
out that his system is not able to align constructs horizontally, as
for example in:
 %
\index{Horizontal spacing}\index{Spacing!horizontal}\index{Blaschek, G.}
 %
\begin{outp}
  \res{with} \id{stmt} $\hat{ }$ \res{do}\myindent\myforce
    \begin{tabular}{ll}
      \id{position} & \ass\ \id{pos} \\
      \id{decLabel} & \ass\ \id{label} \\
      \id{next}     & \ass\ \res{nil} \\
    \end{tabular}
    \myforce
  \res{end}\myforce
\end{outp}
 %
This is a problem that no prettyprinter to date has mastered in an
automatic fashion.

It is an open question whether these problems could be solved using
intelligent algorithms that learn a layout style by example, like for
instance the `intelligent' prettyprinter by Winter and Cook
\cite{Winter89}. But maybe the worst (and oldest) prettyprinting
problem is how to typeset comments! Papers as early as those of
Mohilner \cite{Mohilner78} and Jackel \cite{Jackel80} address this topic
and the core of the problem surely lies in the nature of comments. The
paper ``Programming languages should NOT have comment statements'' by
Kaelbling \cite{Kaelbling88} and that by Grogono \cite{Grogono89}
strongly argue that at last comments should be treated equally as
parts of the language, not as an add-on that is forgotten at first
point during compilation. Grogono brings this to the point by stating
that ``Comments are second class citizens.'' \cite[p.~80]{Grogono89}
They are mostly allowed anywhere in the source code and prettyprinters
who want to preserve them in the output have severe problems doing
that without destroying the program's format, because they have to be
incorporated into the prettyprinting grammar.

\index{Winter, K.}\index{Cook, C.}\index{Prettyprinter!intelligent}
\index{Intelligent prettyprinter}\index{Typesetting comments}
\index{Comments}\index{Prettyprinting problems!worst and oldest}
\index{Mohilner, Patricia R.}\index{Jackel, M.}
\index{Kaelbling, M. J.}\index{Second class citizens}\index{Equality}
\index{Grogono, P.}\index{Prettyprinting grammar!and comments}

There have been attempts to do this in a structured way, i.e.~to
divide comments into classes and to treat each class differently
\cite{Richard77,Rubin83,Fritzson83,Rorbillard89}. Rose and Welsh
distinguish between pre- and postcomments and state that comments must
move with the syntactic elements that they refer to
\cite[p.~660]{Rose81}. Kaelbling even goes as far as to demand ``scoped
comments'', i.e.~comments that explicitly show to which part of the
program they belong.

\index{Rose, G. A.}\index{Welsh, J.}\index{Classes of comments}
\index{Comments}\index{precomments@``precomments'' (Rose and Welsh)}
\index{postcomments@``postcomments'' (Rose and Welsh)}
\index{Kaelbling, M. J.}\index{scoped comments@``scoped comments'' (Kaelbling)}

All these attempts have the same goal, but the problem of prettyprinting
comments will only be solved, if they are treated as an equal part of
a programming language and thus are included in the reference grammar.



















% ######################################################################

\chapter{Future Work}
\label{chap:future:work}

\begin{quote}

``In automatic formatting one should avoid interpreting layout features
of the source text unless they are so special that the user can always
avoid supplying such features inadvertently (e.g., one could
imagine blank lines in the program code being automatically converted
to `\verb"@#"', but even there some programmers may feel that this cramps
their input style). [...] it would be a good thing however if any
popular layout style could be selected as an option to the
prettyprinter (just like \LaTeX{} allows selection of a document style
independently from its contents) [...]''

Marc van Leeuwen \cite{vanLeeuwen95:pp:layout:style}

\end{quote}

\index{Leeuwen@van Leeuwen, Marc}

\noindent After having finally implemented and tested the \pretzel\
program I have noticed, that the program doesn't do very much at all,
although it suits the specifications that belonged to this project. To
enjoy beautifully formatted source code you still have to construct
your prettyprinting grammar. As is seems, the work of fine tuning such
a grammar that the prettyprinter is able to handle the last formatting
detail you desire is quite tedious and time-consuming, especially if
you start to build your grammar from scratch. But after constructing
such a grammar for \PASCAL\ and applying it to a few everyday-example
\PASCAL\ source codes, I was astonished how easy it was to change the
looks of the prettyprinted text. I think that starting with a formal
grammar of your favourite language and trying to enhance it and
transform it into a prettyprinting grammar is surely a better way to
reach your goal. I suppose that the \pretzel\ program could be the
right tool to help you with this task.

\index{Omissions}\index{Prettyprinting grammar}\index{pascal@\PASCAL}
\index{pretzel@\pretzel}

However, since only few people have used \pretzel\ until today there
will surely be a lot of things people miss when using the program.
Here are a few things I have thought of already:

\begin{description}

\item[Allow \t{/*}\dots\t{*/} comments.]  In places where the user has
  an empty production in the formatted grammar file or an empty token
  definition in the formatted token file file, it would be nice to
  have a C like commenting feature with opening and closing
  delimiters. The `\t{//}' comment delimiter isn't nice if you want to
  add an attribute definition to an empty production.

  \index{Empty tokens}\index{-@\t{/*}\dots\t{*/} comments}
  \index{--@\t{//} comments}

\item[Copy comments.]  It surely would make the generated flex and
  Bison files more readable, if \pretzel\ would copy the comments from
  the formatted token and the formatted grammar files into them.

  \index{Copy comments}

\item[Generate \t{\%token} definitions.]  If one would impose the
  restriction to use only uppercase identifiers as terminal token
  names \pretzel\ could automalically generate the list of \t{\%token}
  definitions needed to identify terminals in the formatted grammar
  file.

  \index{token definitions@\t{\%token} definitions}


\item[Change prettyprinting grammar format.] It would be nice to
  be able to insert format commands directly into the grammar rules,
  like for example in:

\begin{inp}
  WHILE expr DO \fmt{indent} stmt \fmt{outdent} \fmt{force}
  $\longrightarrow$ stmt
\end{inp}

This is a point that effects fundamentals of the \id{build\_pparse}
function. Other parts of \pretzel\ are not involved. This is a change
that would be good for new users, since this way of specifying the
grammar is much more intuitive.

\index{Inituitive grammar format}\index{build pparse@\id{build\_pparse}}
\index{Prettyprinting grammar}

\item[Include files for grammars.] As parts of prettyprinting
  grammars occur frequently in many different programming languages
  (such as the formatting of expressions) it would be nice to be able
  to include files that contain these definitions with a simple
  command in the formatted token and formatted grammar files. On a
  more abstract level one could think of a notion of modules of
  grammars, i.e.~parts of a grammar that can be called with arguments
  to suit local demands (``grammar templates''?). But it is still an
  unanswered question whether this is practical, because of the lack
  of method to specify the interface of such a module.

  \index{Include files for grammars}\index{Prettyprinting
    grammar!include files}

\item[Enhance \noweb{} support.] This includes managing multiple
  prettyprinting filters and automatically selecting a filter for the
  right language (if you mix languages in \noweb{} files). The
  \noweb{} support seems the place where the most work can be done to
  get a practical system.

\end{description}

\noindent Suggestions are always welcome.








% ######################################################################

\chapter{Reference}


\begin{quote}

  ``I used to think that prettyprinting was the cats meow, but after
  becoming a bit accustomed to \noweb{} I find that a \CWEB{} printout
  looks somewhat like gibberish.''

  Barry Schwartz \cite{schwartz94:use:of:prettyprinting}

\end{quote}

\index{Schwartz, Barry}

\noindent This chapter contains a complete and detailed reference
of the \pretzel{} system. This is information that mostly is also
contained in the manual pages \t{pretzel(1)} and \t{pretzel-it(1)}.



\section{The Concept of Pretzel}

The concept of \pretzel\ is visualized in figure \ref{concept}.  From
a formal formatting description of a programming language $P$,
\pretzel\ generates a prettyprinting function that can be used to
prettyprint source code in $P$. To get the actual prettyprinting
program, you only have to supply a main program (or use the one that
comes with \pretzel). If the way of prettyprinting needs to be
changed, the user only needs to change formal parts in the input and
use \pretzel\ again to get an enhanced prettyprinter.

\index{pretzel@\pretzel!concept}
\index{P@$P$}


\begin{figure}
\begin{center}

\setlength{\unitlength}{0.012500in}%

\begin{picture}(203,87)(8,744)
\put( 90,789){\makebox(0,0)[b]{pretzel}}
\thinlines
\put( 54,744){\framebox(72,18){}}
\put( 87,777){\vector( 0,-1){  9}}
\thicklines
\put( 30,753){\vector( 1, 0){ 18}}
\put(129,753){\vector( 1, 0){ 24}}
\thinlines
\put( 66,783){\framebox(45,18){}}
\put( 87,819){\vector( 0,-1){ 12}}
\put( 90,822){\makebox(0,0)[b]{formal description}}
\put( 90,750){\makebox(0,0)[b]{prettyprinter}}
\put( 18,750){\makebox(0,0)[b]{Text}}
\put(183,759){\makebox(0,0)[b]{prettyprinted}}
\put(177,744){\makebox(0,0)[b]{text}}
\end{picture}

\end{center}

\caption{The concept of \pretzel.}
\label{concept}

\end{figure}


Figure \ref{detailed-concept} on page \pageref{detailed-concept} shows
the generating process in slightly more detail. The figure shows that
the program \pretzel\ doesn't generate a prettyprinter immediately;
what it does is to produce code that can be turned into a \CPP\
prettyprinting class with the help of two tools:
 %
\index{c++@\CPP}
 %
\begin{description}

\item[flex] A POSIX.2 compliant lexical analyser generator.

\item[Bison] A POSIX.2 compliant parser generator. The version
used with \pretzel\ has to produce \CPP\ compliant source.

\end{description}

\index{flex}\index{Bison}\index{POSIX}

\noindent Users that are already familiar with the basics of both
tools will find it easier to read the following text because the way
of specifying input to \pretzel\ is quite close to the ways used by
them. However, this text aims at people who don't have such knowledge
and will explain everything that is necessary.

Also visible from this figure is that \pretzel\ creates two things:

\begin{itemize}

\item A prettyprinting scanner class and

\item a prettyprinting parser class (containing the actual prettyprinter).

\end{itemize}

\index{Prettprinting scanner}\index{Prettyprinting parser}

The prettyprinting parser is the actual prettyprinter function that you
normally want to get when using \pretzel. The prettyprinting scanner is
used by the parser and takes care of the token formatting. Both
parts are separate modules with a well defined interface and can be
used individually. Hence, you can write your own scanner or parser if
you really need features that the ones produced by \pretzel\ don't have.




\subsection{The Input Files}

The word ``pretty'' is very subjective and so programming language
source code can be printed in a ``pretty'' kind of fashion in almost
infinitely many ways, according to the taste and preferences of the
user. If the task of prettyprinting is handed over to the computer,
the user must specify in detail all of his wishes towards the
appearance of the output of the prettyprinter.

To generate a prettyprinter, \pretzel\ needs two descriptions that it
expects to find in two different files:

\begin{description}

\item[A formatted token file.] This file should contain all information
  concerning the individual formatting of tokens (i.e.~identifiers,
  reserved words, etc.).

\item[A formatted grammar file.] This file should contain a prettyprinting
  grammar, i.e.~a context free grammar enhanced with formatting commands.
  This information is necessary to handle the more global aspects of
  prettyprinting that need information about the language context
  (i.e.~indentation, line breaks, etc.).

\end{description}

\index{Formatted token file}\index{Formatted grammar file}

\noindent The name of the first file might be a little misleading, since
it doesn't contain formatted tokens. Instead it tells you what you will
get from the definitions therein. A name like ``token formatting file''
might have been a little better, but the similarity in names between the
two input files sounded good and seemed to outplay this small inconsistency.

The special formats of these two files are described later in section
\ref{file-formats}. The interface to the generated \pretzel{} classes
is described in detail in section \ref{sec:pretzel:interface}.


\begin{figure}[htb]
  \begin{center}
    \leavevmode
    \input{../figures/detailed-concept.latex}
  \end{center}
  \caption{The current process of generating a prettyprinter with \pretzel.}
  \label{detailed-concept}
\end{figure}




\section{The Format of the Input Files}
\label{file-formats}

\index{File formats}

Now we turn to a vital thing for the user of \pretzel: The format of
the \pretzel\ input files. Here the user has to put a formal description
of the underlying programming language and (with the help of format
commands) state how its text should be formatted.

We have seen that \pretzel\ takes two input files: The formatted token
file and the formatted grammar file. We suggest to use the suffix
`{\tt .ft}' for a formatted token file and `{\tt .fg}' for a formatted
grammar file.

\index{ft@\t{.ft} suffix}\index{fg@\t{.fg} suffix}
\index{Filename conventions}\index{Conventions}



\subsection{The Formatted Token File}

\index{Formatted token file}

The formatted token file contains a list of token definitions with their
corresponding ``prettyprinted'' form. The prettyprinted form of a
token will be called an {\em attribute} or a {\em translation}.

The general outline of the formatted token file is
 %
\begin{outp}
\id{declarations}

\t{\%\%}

\id{token definitions}
\end{outp}
 %
Normally, the {\em declarations} part is empty. You can put a general
description of the file here (as a \C\ comment) and redefinitions of
the default interface go here as well (see section
\ref{sec:pretzel:interface} for more).

The {\em token definitions} section of the formatted token file contains a
series of token definitions of the form:
 %
\begin{outp}
\id{pattern}\qquad \id{token}\qquad \{\id{attribute}\}
\end{outp}
 %
The {\em pattern} must be a valid regular expression (in terms of {\tt
  flex}) and must be unindented. The {\em token} specifies the
symbolic name of the token for the pattern and begins at the first
non-whitespace character after the pattern. The token name must be a
legal name for an identifier in \PASCAL\ notation and {\em must be all
  in upper case}. (Underlines are allowed but not at the beginning of
a word.)

\index{pascal@\PASCAL}

The {\em attribute} for this token, that is it's prettyprinted form,
consists of all text between the two curling brackets `\{' and
`\}'. Attributes can be either simple strings (surrounded by double
quotes) or format commands (like \fmt{force}, \fmt{indent}) or a
combination of both joined together by an optional `\t{+}' sign. Attribute
definitions can cover several lines and the starting `\{' needn't
stand on the same line as the token definition; however subsequent
lines must be indented with at least one blank or one tab. Attributes
can also contain \C{} code. See section \ref{sec:adding:code} or the
manual page \t{pretzel(1)} for details.

% this info belongs here too XXX

If you define strings as part of an attribute definition, you have
to specify them in a \C\ kind of fashion, i.e.~you can insert
newlines and tabs with `\verb+\n+' and `\verb+\t+'. But if you want
to insert a backslash into a string, you mustn't forget to put two
backslashes (`\verb+\\+') into the input file. This is especially
noteworthy if you are using \TeX\ as typesetter, because \TeX\
uses a backslash as a prefix for typesetting commands.

If the definition of the attribute is omitted \pretzel\ creates an
attribute for this pattern by default. The default attribute
consists of the string containing the text matched by the
corresponding pattern.

The user himself may also refer to the matched text by using the
sequence `\t{**}'. Thus
 %
\begin{verbatim}
  "foo"       BAR
  "foo"       BAR     { ** }
  "foo"       BAR     { "foo" }
\end{verbatim}
 %
all have the same meaning.

You can use a `\t{|}' sign as a token name; this signals that the
current regular expression has the same token name (and also the same
attribute) as the token specified in the following line (empty lines
are ignored). An attribute definition behind a `\t{|}' is illegal.
However you may specify regular expressions with neither a token name
nor an attribute to give a default rule or to eat up whitespace.

The following examples are all legal token definitions (and please
note the dot in the very last line):

\begin{verbatim}
  [0-9]           DIGIT

  "{"             OPEN           { "\\{" indent force }

  [a-z][a-z0-9]*  ID             { "{\\it " + ** + "}" }

  "function"      |
  "procedure"     PROC_INTRO     { big_force ** }

  [\t\ \n]        |
  .
\end{verbatim}


The declarations and the token definitions must be separated by a
line containing only the two characters \t{\%\%}. So the shortest
possible formatted token file is
 %
\begin{verbatim}
   %%
\end{verbatim}
 %
but this doesn't seem of any use, does it?




\subsection{The Formatted Grammar File}
\label{fg-file}

\index{Formatted grammar file}

In the formatted grammar file the user encodes the general
prettyprinting grammar for the programming language. This is done by
specifying a context free grammar of the language and by adding
information about the creation of new attributes in every rule.

\index{Context free grammar}

The formatted grammar file is the second and last input to the \pretzel\
program. Its general outline looks like this:

\begin{outp}
\id{token declarations}

\t{\%\%}

\id{grammar rules}
\end{outp}

The {\em token declarations} section may be empty and the separator
between the two parts of the file (\t{\%\%}) must appear
unindented on a single line by itself. Before we look at these
declarations, let's have a look at the grammar rules.

The {\em grammar rules} section contains the collection of rules of
the context free grammar that can be accompanied by an attribute
definition.  A rule is specified by stating the resulting token, a
colon and then the series of tokens which will be reduced by this
rule. The rule is ended by a semicolon. A block definition in \PASCAL\
for example might look like this:
 %
\begin{verbatim}
  block : BEGIN stmt_list END
  ;
\end{verbatim}

Following the token list on the right side of the colon can be an
attribute definition; this definition states, how the translation of
the produced symbol is obtained from the tokens on the right side of
the rule.

An attribute definition is bracketed amidst curling brackets `\{'
and `\}' and can again consist of strings (in double quotes) and
format commands or both joined together with `\t{+}'. But here you
can also refer to the attributes of the tokens on the right side
of the rule. This is done in a slightly awkward notation with a number
that is preceded with a `\t{\$}' dollar sign. The numbers refer to
the order of appearance of the symbols on the right side of the
rule. So `\t{\$1}' refers to the first token of the rule,
`\t{\$2}' to the second, \dots

Again attribute definitions are allowed to span several lines and
strings must be specified in \C\ manner. They can also contain \C{}
code as described in section \ref{sec:adding:code}.

\index{c@\C}

For example, here again is the possible definition of a block in
\PASCAL, now with an example attribute definition:
 %
\begin{verbatim}
  block : BEGIN stmt_list END   { $1 + $2 + force + $3 }
  ;
\end{verbatim}

\index{pascal@\PASCAL}

The attribute of a \t{block} will therefore consist of the
attributes of the \t{BEGIN} and \t{stmt\_list} tokens, joined
together with a \fmt{force} command and the translation of the
\t{END} token.

The attribute definition may be omitted. If this is so, \pretzel
will by default form the attribute of the produced symbol from the
simple concatenation of the attributes on the right side of the
rule. For instance
 %
\begin{verbatim}
  stmt : block SEMI
  ;
\end{verbatim}
 %
means the same as:
 %
\begin{verbatim}
  stmt : block SEMI       { $1 + $2 }
  ;
\end{verbatim}

Of course you may also have empty right sides of a rule (to produce
things out of nothing) or simply concatenate two or more rules
resulting in the same symbol with a `\t{|}'. So the following are
legal rules:
 %
\begin{verbatim}
  stmt_list   :                      { force }
              | stmt_list stmt SEMI  { $1 $2 $3 force }
  ;
\end{verbatim}


To end this subsection, we have to return to the {\em token
declarations} section of the formatted grammar file. Here we have to insert
a special line for every terminal token that appears in the grammar
rules. These definitions are of the form `\t{\%token} \id{tokenname}'.
This part of the formatted grammar file is owed to Bison and should
be removed in subsequent versions.



\subsection{Comments and Code}

\index{Comments!in \pretzel\ input files}

There is a very simple way of putting comments into the formatted
token and formatted grammar files. This is done in a \CPP\ kind of
manner by preceding the comment with a double slash (`\t{//}'). All
characters between this sign and the end of the line are ignored by
pretzel.

\index{c++@\CPP}

In both files you can put additional \CPP\ code before and after the
definitions/grammar sections.  If you want to insert code at the end
of your file, you have to put a second `\t{\%\%}' on a line by itself
and put the code behind it. \C/\CPP\ Code before the definitions/rules
section has to be tied in with a `\%\{', `\%\}' pair. Inserting extra
code is interesting for people who want to call this code from within
the attribute definitions. See section \ref{sec:adding:code} for
details.

% should be put here XXX



\section{Synopsis of \t{pretzel} and \t{pretzel-it}}

\subsection{\t{pretzel-it}}

The shell script \t{pretzel-it} uses \pretzel{} to build a simple
prettyprinter executable. It minimizes building a \pretzel{}
prettyprinter to just one shell command.

You have to provide the same two input files to \t{pretzel-it} as to
\pretzel. These two files are called the formatted token file (suffix
\t{.ft}) and the formatted grammar file (suffix \t{.fg}).  Both files
need to have the same prefix.  From this input, \t{pretzel-it}
generates an executable prettyprinter.

To get to know the options, type
 %
\begin{verbatim}
  pretzel-it -h
\end{verbatim}
 %
at the command line. The full usage is:
 %
\begin{verbatim}
  pretzel-it [-iqvdnh] language ppname
\end{verbatim}
 %
Here's an explanation of the options:
 %
\begin{description}

  \item[-i]   Don't remove intermediate products of pretzeling.
  \item[-q]   Run quietly.
  \item[-v] Verbose mode, print shell commands before invoking (for
    debugging).
  \item[-d] Turn prettyprinter debugging features on by default (for
    debugging the prettyprinting grammar).
  \item[-h]   Print full usage message.
  \item[-n] Noweb mode, will produce a prettyprinting filter
    \t{ppname} compatible to Norman Ramsey's \noweb{} literate
    programming system. The filter can be inserted into the \noweb{}
    pipeline using \t{noweave}'s \t{-filter} option.

\end{description}
 %
See also the manpage \t{pretzel-it} and chapters \ref{chap:using:pretzel}
and \ref{chap:pretzel:meets:noweb}.



\subsection{\t{pretzel}}


\pretzel\ is invoked by typing
 %
\begin{verbatim}
  pretzel
\end{verbatim}
 %
at the command line. The full usage of \pretzel\ can be
obtained using the \t{-h} option. It is:
 %
\begin{verbatim}
  pretzel [-qtgdh] [-o outfile] (prefix | file1 file2)
\end{verbatim}
 %
Here's an explanation of the options:
 %
\begin{description}

\item[-q] Run quietly (no screen output).

\item[-t] Process formatted token file only.

\item[-g] Process formatted grammar file only.

\item[-d] Run in debug mode (i.e.~print out debugging information
on the screen while running).

\item[-h] Show full usage.

\item[-o outfile] Names of the produced output files begin with ``outfile''.

\end{description}
  %
The options \t{-t} and \t{-g} are mutually exclusive, i.e.~you
can't choose both at the same time.

\index{pretzel@\pretzel!options}

The command line parameters have different meanings depending on
whether one ore two names are given. If there is only one parameter,
it specifies the prefix of both formatted token and formatted grammar
files. The suffixes `{\tt .ft}' and `{\tt .fg}' are assumed. But if
there are two parameters at the command line, \pretzel\ will take the
first as full name of the formatted token file and the second as full
name of the formatted grammar file. In this case, the output files
will get a default name. The output files will have endings `{\tt .l}'
(token file) and `{\tt .y}' (grammar file).

\index{ft@\t{.ft} suffix}
\index{fg@\t{.fg} suffix}
\index{l@\t{.l} suffix}
\index{y@\t{.y} suffix}




% ######################################################################

\addcontentsline{toc}{chapter}{Bibliography}
\bibliographystyle{plain}
\bibliography{../bibliographies/prettyprinting,../bibliographies/litprog,../bibliographies/supplemental}


% ######################################################################

\addcontentsline{toc}{chapter}{Index}
\printindex


\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

$Log: pretzelbook.tex,v $
Revision 2.1  1997/07/09 14:39:29  gaertner
Updates for 2.0g: this is second edition. Cahpetr on noweb
enhanced, built in info on Java prettyprinter.

Revision 2.0  1996/12/16 17:41:38  gaertner
finished version for 2.0

Revision 1.2  1996/12/16 13:35:15  gaertner
Revised version. Only reference and future work missing.








