% This is a file written using noweb
%
% $Id: fgscan.nw,v 1.4 1996/09/12 18:43:53 gaertner Exp $
%

\documentclass{article}
\usepackage{a4,version,pretzeldoc,noweb}
\begin{document}

\includeversion{bigspecs}
%\includeversion{impls}
%\excludeversion{bigspecs}% without elaborate specification
\excludeversion{impls}% without implementation



\title{A Scanner for Formatted Grammar Files}
\author{Felix G\"artner}
\maketitle


\begin{abstract}
    This document describes the interface \begin{impls}and
    implementation\end{impls} of a scanner for formatted grammar
    files. A formatted grammar file is a file containing a description
    of a grammar for prettyprinting a certain programming language
    which is enhanced with format commands. It is used within the
    framework of the ``pretzel'' project. Pretzel is a universal
    prettyprinter generator.
\end{abstract}


\tableofcontents


\section{Introduction}

A {\em prettyprinting parser} is a parser that does prettyprinting
as a result of parsing. Such a parser normally uses an attribute
grammar, i.e.~every token has an associated attribute. These attributes
specify how the token looks. An attribute grammar that contains such
prettyprinting information will be called a {\em formatted grammar}.
Together with information about the formatting of terminal tokens
(so called {\em formatted tokens}) a formatted grammar defines a
prettyprinter.

The ``pretzel'' project aims to generate such a prettyprinter
automatically from formatted token and formatted grammar description
laid down in files of a special format. These files are called the {\em
formatted tokens file} and the {\em formatted grammar file}. The
precise format of these files can be obtained by reading the
main documentation of the pretzel project. I will assume that you
know about this and also about the prettyprinting method that the
generated prettyprinter uses.

This document describes the interface \begin{impls}and
implementation\end{impls} of a program that is used to generate such a
prettyprinting parser. It is a scanner to scan formatted grammar
files. This scanner is used by the actual generator routine that
creates the prettyprinting parser.

This is an overview over the frame of the header file for the scanner:

<<header>>=
#ifndef FGSCAN_H
#define FGSCAN_H

<<Header files to include in the specification>>
<<Declaration of the formatted grammar file scanner class>>

#endif
@

\begin{impls}

  The implementation will use the flex scanner generator (installed on
  the development site is version 2.4.6). We won't use the feature of
  flex to create a \CPP\ scanner class, because we want to stick with
  stuff that has been tested and that will probably work. A flex file
  has the following frame.

<<flex-implementation>>=
%{
#include"fgscan.h"    /* include own header */
%}

<<flex definitions>>
%%
<<flex rules>>
%%
<<flex user code>>

@

\end{impls}


\section{Interface (Overview)}

This section describes the interface of the scanner that scans
formatted grammar files. Formatted grammar files contain a context
free grammar which is enhanced by formatting instructions. A formatted
grammar file is used to build a prettyprinter for the so specified
programming language.

There are three things we want to do with the scanner:
\begin{itemize}
\item Create a new scanner and initialize it to tell it where to take
  the input characters from.
\item Call it, to know, which token is next in the input stream (this
  is called ({\em scanning}).
\item Destroy the scanner explicitly (to free memory, clean up or get
  ready for a new formatted grammar file).
\end{itemize}

Here's the declaration of the new scanner class.

<<Declaration of the formatted grammar file scanner class>>=
class Fgscanner {
public:
  Fgscanner(istream*);    /* create and initialize new scanner */
  ~Fgscanner();           /* remove a scanner */
  int scan(Attribute**);  /* scan for a new token */

  int line();             /* returns line number of last token scanned */

  <<private members of the new scanner class>>
};

@ The function [[line]] is an additional member function aimed to help
debugging.

The [[scan]] member function returns the token code of the next token
together with a textual description of the details as a reference
argument of type [[Attribute*]]. See the {\tt attr.nw} file for a
description of this class. Anyway, we have to include these headers.

<<Header files to include in the specification>>=
#include"attr.h"       /* the Attribute class */
#include<iostream>

using namespace std;
@




\section{Interface (Details)}


In this section I will present every member function of the new
scanner class in a detailed fashing for a complete
reference\begin{impls} and will explain the ideas and details behind
  the implementation\end{impls}.


\begin{impls}

  A scanner generated by flex is mainly a function called [[yylex]]
  that scans from a given [[FILE*]] and returns token codes as
  integers. We'll use this function to scan the formatted token file
  and to return an additional string with the token code that describe
  the token in more detail and can be used to piece together the
  target file implementing the prettyprinting scanner. We'll use the
  [[Attribute]] class for this purpose. (Though it wasn't designed for
  it, we can still use it here.) We also have to redefine the
  interface for [[yylex]] to suit these requirements.

<<flex definitions>>=
%{
#undef YY_DECL
#define YY_DECL int yylex(Attribute* *rvalue)
%}
@

  By default, the name of the scanner function is [[yylex]]. By
  specifying the {\tt -P} option at the command line you can redefine
  the ``yy'' prefix and thus change the name of the scanner function.
  The implementation expects that the scanner is generated using the
  option ``{tt -PPretzel_fg}'' and thus will expect the scanner
  function to be called:
   %
  \begin{quote}
    [[Pretzel_fglex]]
  \end{quote}

\end{impls}

A short word about using multiple scanners within one program: Because
of drawbacks of the underlying algorithm \begin{impls}(the interface
of flex)\end{impls} the resulting scanner is not reentrant and using
multiple instances of the scanner class in the same program will be
extremely hazardrous. This might seem a severe drawback, but isn't too
bad in the end, because it was built to be used in only one instance.
\begin{impls}This is also the reason why I shouldn't have used a class
to implement it, but the interface was designed before learning about
flex' deficiencies.\end{impls}

\begin{impls}

  Because of the above note, we'll use static variables instead of
  private members to handle the private information of the class.

<<private members of the new scanner class>>=
/* none */
@

  We want the scanner to read from an input stream and not from a
  [[FILE*]], so we have to take care about this. We'll define a
  static variable that holds the [[istream]] object pointer and will
  be used by the scanner.

<<flex definitions>>=
%{
static istream* yyin_stream;
%}
@

  Also, we have to modify the way, in which [[yylex]] reads it's
  data. This is done by redefining the [[YY_INPUT]] macro. Thanks to
  Christine Detig for the hint.

<<flex definitions>>=
%{
#undef YY_INPUT
#define YY_INPUT(buf, result, max_size) {               \
        if (yyin_stream->eof()) result=YY_NULL;         \
        else {                                          \
                yyin_stream->read(buf, max_size);       \
                result=yyin_stream->gcount();           \
        }                                               \
}
%}
@

  The current line number has to be stored somewhere too. This is done
  within the static variable [[line]].

<<flex definitions>>=
%{
static int line_number = 1;
%}
@

\end{impls}




\subsection{Creating the new Scanner}

The constructor of the [[Fgscanner]] class creates a new scanner for
formatted grammar files. You have to specify an [[istream]] pointer,
which is the stream from which the tokens are read.

\begin{impls}

  The implementation of the constructor is pretty easy. It simply
  stores the given input stream pointer and resets the line number
  count. The line number is initially set to 1 since we have no line 0.

<<flex user code>>=
Fgscanner::Fgscanner(istream* is)
{
  line_number = 1;
  yyin_stream = is;
}

@

\end{impls}


\subsection{Destroying a Scanner}

The [[~Fgscanner]] simply destroys the specified scanner again. An
[[Fgscanner]] is a one time use object. To start reading from a new
input stream you should first destroy the old scanner and then create
a new one using the constructor.

\begin{impls}
  This is implemented by resetting the internal scanner in case we
  want to scan a new formatted token file. Note that although the
  scanner object dissappears, the internal scanner state still is
  valid.

<<flex user code>>=
Fgscanner::~Fgscanner()
{
  yyrestart(NULL);
}

@
\end{impls}




\subsection{Scanning}

The next function is the normal scanner member function called
[[scan]]. It returns token codes as integer values together with the
corresponding attribute (as a call by reference argument). Table
\ref{tab:returnvalues} shows all the possible values that [[scan]]
might return. They are partly defined as \C\ macros, and partly simply
the character values of the given symbols. The macros are defined in a
header file that is generated by the parser for formatted token files
(the file {\tt fgramdefs.h}). See the specifications in {\tt
fgparse.nw} for details.

From version 2.0 onwards \pretzel{} allows inclusion of code within
attribute definitions. There are three additional tokens that the
scanner detects: [[BEG_CODE]], [[CODE]], and [[END_CODE]]. They refer
to the starting code delimiter within attribute definitions (usually
a ``{\tt ]}''), a code line within, and the corresponding closing
delimiter.


\begin{impls}
  The implementation has to include this header file.

<<flex definitions>>=
%{
#include "fgramdefs.h"
%}

@

\end{impls}


\begin{table}[htbp]
  \begin{center}
    \leavevmode
    \begin{tabular}{IlIp{5cm}I}
\hline
token & comment \\
\hline
\verb+PERCENT_PERCENT+  & A line containing only `\verb+%%+'. \\
\verb+A_LINE+           & A line from the definitions part or
        the \C\ code part. \\
\verb+DOLLARS+          & Any symbol from `\verb+$1+' to `\verb+$9+'. \\
\verb+STRING+           & A string from an attribute definition. \\
\verb+{+                & Start of attribute definition. \\
\verb+}+                & End of attribute definition. \\
\verb#+#                & Concatenation inside attribute definition. \\
\verb+;+                & Semicolon. \\
\verb+:+                & Colon. \\
\verb+I+                & Vertical line. \\
[[BEG_CODE]]            & starting code delimiter in attribute definitions \\
[[CODE]]                & a line of code in attribute definitions \\
[[END_CODE]]            & ending code delimiter in attribute definitions \\
\verb+FMTCMD+           & A format command. \\
\verb+ID+               & A token identifier. \\
\verb+UNKNOWN+          & A set of unmatched characters. \\
\hline
    \end{tabular}
    \caption{(Symbolic) return values of the [[scan]] member function.}
    \label{tab:returnvalues}
  \end{center}
\end{table}

Basicly a formatted grammar file contains a prelude of definitions and
\C\ code which is followed by a line containing ``\verb+%%+''. Then
follow a set of grammar rules which are enhanced by formatting
instructions. Optionally there may be an additional line with
``\verb+%%+'' followed by \C\ code.

The [[scan]] member function returns the semantic value (i.e.~part of
the text to build the prettyprinting parser) as a call by reference
argument.

\begin{impls}

  The [[scan]] member function is a delayed call to [[yylex]].

<<flex user code>>=
int Fgscanner::scan(Attribute* *rvalue)
{
    return(yylex(rvalue));
}
@

The exact behaviour, i.e.~a complete specification of the flex
regular expressions can be found in section \ref{sec:flexbody} on page
\pageref{sec:flexbody}.

\end{impls}


\subsection{Debugging}

There is an additional member function that can be used for debugging
purposes. The value of [[line()]] returns the line number that the token
was in that was last returned.

\begin{impls}

  Looking at the implementation we have to note that the internal
  variable has to be incremented every time we scan a newline in the
  input file. This has to be checked in the definitions of section
  \ref{sec:flexbody} and is a possible bug source!

<<flex user code>>=
int Fgscanner::line()
{
  return(line_number);
}
@

\end{impls}














\begin{impls}

\section{The flex Token descriptions}
\label{sec:flexbody}

This section contains the complete flex implementation of the scanner
class. Note that we have to increment the current line number every
time we encounter and scan a newline.


The really important things in the formatted grammar file are the lines
inbetween the \verb+%%+ tokens (or between the first and the end of
the file if no user code is given). So we'll have the scanner ignore
everything from the beginning of the file up to the first two percents
and then start it's real work. We'll distinguish between these two
states using an exclusive start condition [[R]] (for ``rules'').

<<flex definitions>>=
%x R
@

Now lets have a look on how the scanner reads the top lines and gets
into this start condition.

When the scanner finds the first `\verb+%%+', he'll return a
[[PERCENT_PERCENT]] token and switch into ``rules'' mode. Until he
finds such a line, he'll keep on returning the prelude of the file
line by line.

<<flex rules>>=
^"%%"\n         {*rvalue=create(yytext);
                BEGIN(R); line_number++; return PERCENT_PERCENT;}

.*\n            *rvalue=create(yytext); line_number++; return A_LINE;
@

So let's assume we are in ``rules mode''. Then we basicly know what we
can expect. These are the `reserved words':

<<flex rules>>=
  /* references to semantic values of tokens */
<R>"$"[1-9]     *rvalue=create(yytext); return DOLLARS;

  /* strings */
<R>\"[^\"]*\"   *rvalue=create(yytext); return STRING;

  /* special delimiters */
<R>"{"          |
<R>"}"          |
<R>"+"          |
<R>";"          |
<R>":"          |
<R>"|"          *rvalue=create(yytext); return yytext[0];

@

Before we continue with looking at the format commands, here's the
stuff needed for version 2.0 of \pretzel{}. We need to detect code
parts within  attribute definitions. The opeing and closing delimiters
for code will be angled brackets. If we encounter such an opening
delimiter, we have to return all code up to the closing delimiter line
by line. Because this is again a different context of speach, we need
an additional (exclusive) start condition. It will be called [[AC]]
for ``attribute code''.

<<flex definitions>>=
%x AC
@

<<flex rules>>=
<R>"["           *rvalue=create(yytext); BEGIN(AC); return BEG_CODE;

@ Before we continue, a short thought: Say, we've seen an opening code
delimiter and we're in [[AC]] mode. Now all lines must be returned up
to the point where we find a matching closing delimiter. Closing
delimiters (usually an angled bracket) must be escaped if they are
used within the code. The escape character is stripped off when the
string is returned using the [[unescaped]] function.

<<flex rules>>=
<AC>(([^\]\n])|(\\\]))*         { *rvalue=create(unescaped(yytext));
				return CODE; }

<AC>(([^\]\n])|(\\\]))*\n       { *rvalue=create(unescaped(yytext));
				line_number++; return CODE; }

<AC>"]"         *rvalue=NULL; BEGIN(R); return END_CODE;

@ The [[unescaped]] function removes backslashes from a given string if
they are followed by a closing delimiter (currently a `{\tt ]}'). The
string length shrinks by the number of backslashes removed. [The same
routine appears in the file {\tt ftscan.nw}. That's why it's declared
[[static]]. It should be part of the pretzel library.]

The implementation has two pointers into the array. The first points
to the position where the next unescaped character should be placed,
the second one holds the position of the next character to be
inspected. We only shorten the given string. That's why we don't
[[malloc]] storage and work on a copy.

<<flex definitions>>=
%{
static char* unescaped(char* s)
{
    int left = 0;   /* next position to place [[s[right]]] */
    int right = 0;
    while (s[right] != 0) {  // should be `!= NULL', but some
                             // compilers choke on it
        if ((s[right] == '\\') && (s[right+1] == ']')) {
            right++;
        }
        s[left] = s[right];
        left++; right++;
    }
    s[left] = s[right];  /* terminate string */
    return(s);
}
%}
@

So let's continue now with the format commands we can find.

<<flex rules>>=
  /* format commands */
<R>null                 |
<R>indent               |
<R>outdent              |
<R>opt0                 |
<R>opt1                 |
<R>opt2                 |
<R>opt3                 |
<R>opt4                 |
<R>opt5                 |
<R>opt6                 |
<R>opt7                 |
<R>opt8                 |
<R>opt9                 |
<R>backup               |
<R>break_space          |
<R>force                |
<R>big_force            |
<R>no_indent            |
<R>cancel               *rvalue=create(yytext); return FMTCMD;

  /* abbreviations: */
<R>bsp                  *rvalue=create("break_space"); return FMTCMD;
<R>b_force              *rvalue=create("big_force"); return FMTCMD;

@

Here is the rule for normal token identifiers. Note that only
uppercase is allowed!

<<flex rules>>=
<R>[A-Za-z][A-Za-z0-9_]*      *rvalue=create(yytext); return ID;

@

We eat up whitespace within the rules part and neglext comments.

<<flex rules>>=
<R>[\ \t]                      /* eat up whitespace */
<R>\n           line_number++; /* but count lines */
<R>"//".*                      /* and neglect comments */

@

To finish off the rules, we have to lookout for another two percent
signs in a line by themself

<<flex rules>>=
<R>^"%%"\n      {
                *rvalue=create(yytext);
                BEGIN(INITIAL); line_number++; return PERCENT_PERCENT;
                }

@

For any other character that doesn't match one of the other rules, we
return an [[UNKNOWN]] token.

<<flex rules>>=
<R>.            return(UNKNOWN);
.               return(UNKNOWN);
@




\section{Additional flex functions}

This section contains the definition of the [[yywrap]] function needed
to complete a flex scanner. Returning 1 means that the scanner can
exit when receiving an end of file indication.

The [[extern "C"]] is due to Lee Wittenberg (Thanks!) to get around
linking problems with some versions of gcc.

<<flex definitions>>=
%{
extern "C" int yywrap() {
    return(1);
}
%}
@

\end{impls}


\end{document}


%
% $Log: fgscan.nw,v $
% Revision 1.4  1996/09/12  18:43:53  gaertner
% Added extern "C" (thanks Lee).
%
% Revision 1.4  1996/09/12  18:43:53  gaertner
% Added extern "C" (thanks Lee).
%
% Revision 1.3  1996/09/02  16:03:28  gaertner
% Fixed the `unescaped' routine to work with g++.
%
% Revision 1.2  1996/04/22  09:18:20  gaertner
% Allows user code in attribute definitions now.
%
% Revision 1.1  1996/03/21  08:13:57  gaertner
% Initial revision
%
%


