%
% $Id: chap4.tex,v 1.4 1995/04/20 13:17:00 gaertner Exp gaertner $
%
% $Log: chap4.tex,v $
% Revision 1.4  1995/04/20  13:17:00  gaertner
% Corrections proposed by JS
%
% Revision 1.3  1995/02/23  12:48:00  gaertner
% Final revision (including figures, but without index).
%
% Revision 1.2  1995/02/21  09:30:00  gaertner
% Changed \mkWEAVE and \preFLEX and \preBISON invocations so that
% the text will compile.
%
% Revision 1.1  1995/02/21  09:25:16  gaertner
% Initial revision
%
%



\chapter{The Design of the System}
\label{four}

The idea behind the \pretzel\ program is not new. The way of using a
prettyprinting grammar for formatting and the idea of generating a
\WEB\ system from a formal specifiacation of the underlying programming
language have been presented by Knuth and Ramsey years ago. The new
thing behind the \pretzel\ program is that it combines these two ideas
to produce output that is useable in a lot of different fashions.

This chapter presents the evolution of the \pretzel\ program from its
early design to the history of its implementation.




\section{The Attribute Grammar}

As said earlier, the main working mechanism of the generated
prettyprinter is the use of an attribute grammar to build up
prettyprinted code during a bottom-up parse of the programming
language text. So one of the first steps was to look at a system that
had implemented this already: Knuth's initial \WEAVE\ program and the
\CWEAVE\ program that is derived from it. The first bigger aim in this
project was then to build a prototype prettyprinter that implemented
this notion, using Bison and the semantic actions provided in it to put
together the code using a set of specially designed functions from a
run time library.  In fact, it was the mere design of this run time
library that took up the biggest part of the time in the design stage.


\index{weave@\WEAVE}\index{cweave@\CWEAVE}\index{Knuth, Donald E.}




\subsection{Attributes}

The basic idea that was the starting point of the design was to
provide a recursive data type that implemented attributes or
translations of grammar tokens and could be used during parsing. This
data type was to be called {\em attribute} and should be implemented
using \CPP. An attribute could be recursively defined as follows:

\begin{itemize}

\item A simple string is an attribute.
\item A format command is an attribute.
\item Any combination of two or more attributes is again an attribute.

\end{itemize}

This definition resulted from hand-simulating the process done by
Knuth's \WEAVE\ program on the blackboard and on notepaper. It seemed
at first that it would be a good idea to implement attributes as a
long linked list, because the result of the prettyprinting always
appeared to be a long list of strings and format commands. Whenever a
grammar rule was chosen, the attributes of the tokens were simply
inserted into a new list that represented the attribute of the
resulting token.

For efficiency reasons this idea was soon abandoned and the insertion
of attributes was done by inserting references (visualized as long
arrows on paper) into newly created attributes. The resulting
structures looked very much like trees and the resulting output could
be obtained with a simple tree traversion.

An abstract way of defining attributes using the \CPP\ classes was to
have one abstract base class, of which all of the three attribute
types could be derived from, namely: strings, format commands and
sequences of attributes. This was realized in the \id{Attribute}
class. It provided three kinds of attributes, i.e.~subtypes of
\id{Attribute}:

\begin{enumerate}

\item \id{Text\_attribute} to store simple strings.

\item \id{Format\_attribute} to store format commands (implemented as
an enumeration type).

\item \id{Attributelist} to store a sequence of objects of type
\id{Attribute} (implemented as a linked list holding attribute
pointers).

\end{enumerate}

The recursive implementation is hidden elegantly in the definition of
\id{Attributelist}, because such a list can contain any attribute, a
string, a format command or simply again a list of such attributes.

The final thing concerning this class was to provide a set of
useful functions to handle objects of this type. This set
simply consisted of two functions, namely:
 %
\begin{enumerate}

\item \id{create()} to create a \id{Text\_attribute} or a
\id{Format\_attribute} from a simple string or the name of a
format command.

\item \id{join()} to create an \id{Attributelist} from a sequence
of attributes.

\end{enumerate}
 %
Note that both functions return valid attributes and thus can be
used as arguments to each other. Using attributes in this fashion
therefore creates the trees that appeared earlier on paper or on the
blackboard automatically in the computer's memory.



\subsection{Outputting Attributes}

Creating huge trees of attributes in memory would be useless unless there
would be a possibility to get them out again and output them into a
file or onto screen. As stated earlier this could be done by a simple
tree traversion in pre-order. However, the object oriented approach
that was built into the attributes now seemed to inhibit an easy way
of doing this, since all of the nodes of the tree were simply of type
\id{Attribute} and their initial subtype could not be retieved by
simply looking at them. How do you know how to output an attribute,
when you don't know if it is a string, a format command or even a
list of attributes?

The solution to this problem is really one of the most interesting
parts of the whole program and gets to the actual core of ``real''
object oriented programming. The idea was to provide a public member
function called \id{print} in each subclass of \id{Attribute} that
would automatically output an object of its class in the right way,
provided it was told where to put the output. This was the birth
of the \id{Output} class. Objects of this class (``outputs'') can
be used to output attributes.

An output is attached to a \CPP\ \id{ostream} class object during
creation. Any calls of the \id{print} member functions of attributes
must say into which output they want to send their information. But
the problem of calling the right kind of \id{print} function for every
attribute is obsolete because of the built-in dynamic dispatch: every
object knows its type and can therefore invoke its own specific member
function. (Sending information to an output is done by taking the
string or the format command out of the attribute and calling a
special function \id{do\_output} of the output with this information.)
The traversion is achieved by simply implementing the \id{print}
function of the \id{Attributelist} as a subsequent invocation of the
\id{print} functions of its members.

The output to which this stream of attributes is sent then has full
control over the information flow before it sends it out to the
\id{ostream} object to which it is connected. In fact, an output can
do much more than merely directing information flow to a specific
destination: it may also be used to do any kind of processing of the
attribute information, such as interpreting the format commands and
translating them into typesetter understandable format; or it can do
the preprocessing that is caught by the two phrases on page
\pageref{two-phrases}, i.e.~reducing a sequence of \fmt{force}
commands to just one or handling the behaviour of the \fmt{cancel}
command. The internal structure of each output just has to wait for
subsequent calls of the \id{do\_output} function and handles the
incoming information as it wishes.\footnote{Unfortunately this causes
  the \id{Output} class to neglect the fundamental modularity of
  modern prettyprinters as described in chapter \ref{one}. See
  chapter \ref{five} (future work) for more on this topic.}

Both the \id{Attribute} class and the \id{Output} class are the central
and decisive part of the run time library. Attributes are used to build
up the translations of tokens during parsing; Outputs are used to
interpret and output these attributes to a given output stream.


\section{The Structure of Pretzel}

After designing and implementing the run time library for the
generated prettyprinter all of the functions were tested thoroughly
with a small prototype prettyprinter that could handle simple \C\
code. The main result from this time of testing was that using
attributes and outputs within semantic actions of a Bison grammar to
generate prettyprinted output actually worked.

The next step was to think of a way of specifying a prettyprinting grammar
in machine readable form that would be intuitive and similar enough to
the way used by Knuth in his documentation to \WEAVE. After sketching
out some rough ideas I turned to actually defining Bison grammars to parse
this notation and to translate this into valid flex and Bison files.
This was the birth of the formatted token and the formatted grammar files
and the functions that transform them into valid flex and Bison code.

The general idea of these functions is to transform a sequence like
 %
\begin{verbatim}
    "begin"    BEG    {"\\&{begin}"}
\end{verbatim}
 %
into a line of flex like
 %
\begin{verbatim}
    "begin"   {*rvalue=create("\\&{begin}"); return(BEG);}
\end{verbatim}
 %
where \verb+*rvalue+ stores the semantic value of the token and is
of type \id{Attribute}$\ast$.

A declaration in the formatted grammar file like
 %
\begin{verbatim}
    block  : BEG stmt_list END    {$1 + $2 + force + $3}
    ;
\end{verbatim}
 %
should be transformed into lines of Bison input as in:
 %
\begin{verbatim}
    block  : BEG stmt_list END    {$$=join($1,$2,create(force),$3);}
    ;
\end{verbatim}
 %
It is again an astonishing point to see the universality of the \id{Attribute}
class. Designed for building up attributes during parsing within the
prettyprinter, they suddenly were used to generate the code put into
the flex and Bison files.

The result of all this coding were two generators (see also figure
\ref{generator-funs}):

\begin{itemize}

\item A \id{build\_pscan} function that reads a formatted token file
  from an \id{istream} class object, parses and translates the
  contents of the file and writes to an \id{ostream} class object the
  generated code of a prettyprinting scanner. This scanner acts in the
  way that was specified in the input.

\item A \id{build\_pparse} function that reads a formatted grammar file
  from an \id{istream} and writes a Bison source for a prettyprinting
  parser to an \id{ostream}. This parser acts in the way specified in the
  input.

\end{itemize}

\begin{figure}[htb]
  \begin{center}
    \leavevmode
    \input{../figures/generator-funs.latex}
  \end{center}
  \caption{The two generator functions within \pretzel.}
  \label{generator-funs}
\end{figure}

The translated code not only consists of the pure flex and Bison definitions
for the prettyprinting grammar, but also contains the code for the
actual prettyprinting functions, that were the major goal of this project.
These two functions do the main work of \pretzel.

The last step of this project was then to write the actual \pretzel\ program.
It simply does nothing else but to form a nice user interface to the
two parser functions. See figure \ref{module-depends} for an overview over
the dependency structure of the modules involved in \pretzel.


\begin{figure}[htb]
  \begin{center}
    \leavevmode
    \input{../figures/module-depends.latex}
  \end{center}
  \caption{The dependency structure of the modules of \pretzel.}
  \label{module-depends}
\end{figure}

\section{History}

The design phase began very much in the beginning of 1993 with an
analysis of the sourcecode of Knuth's \PASCAL\ \WEAVE\ program.  In
May 1993 I had designed a prototype prettyprinter for simple \C\ code
that simulated very much the kind of prettyprinter that \pretzel\
should generate. After testing and playing around with it, the second
and last step could begin.

In June 1993 I thought of a way to specify a prettyprinting grammar
that would be easy to read and easy to parse. At this point the
prettyprinting scanners and parsers emerged. As mentioned earlier they
themselves were written in flex and Bison and use the \id{Attribute} class
during code generation.

Finally I had only to write a main program that would invoke
these two parsers and write the generated code into files. I began
coding this program in July 1993 and called it \pretzel. The final
revisions are documented in the \t{diary} file that is part of the
\pretzel\ distribution.

