% noweb source
%
% $Id: nowebpretzelpp.nw,v 1.8 1999/01/19 10:59:54 felix Exp $
%
% History at end
%
\documentclass{article}
\usepackage{noweb}
\usepackage{pretzel-latex}
\begin{document}

\title{\texttt{nowebpretzelpp.nw} \\ A Prettyprinter for {\tt noweb}}
\author{Felix G\"artner}
\maketitle

\tableofcontents

\section{Introduction}

This document describes the interface and implementation of a
prettyprinter for {\tt noweb}. It is written in C++ and conforms to an
API given to me by Norman Ramsey.



\section{Interface}

[This section is the API by Norman Ramsey. Additional text by the
current author is put in angled brackets.]

Here is a suitable API for a prettyprinter for noweb.
Input to be prettyprinted is a sequence of strings, newlines, and
chunks used.  A [[STRING]] is always free of newlines.

<<API for prettyprinter>>=
typedef enum parttype {
  START_OF_CHUNK=1, END_OF_CHUNK, STRING, USE_CHUNK, NEWLINE,
  LITERAL, DEFINITION, WHATSIT
} Parttype;
@

The prettyprinter calls back into the application as follows:

<<API for prettyprinter>>=
typedef void (*PrettyPrinterCallback) (void *closure, Parttype type, char *contents);
@

The [[contents]] must be
\begin{quote}
\begin{tabular}{ll}
[[START_OF_CHUNK]]&The name of the chunk being prettyprinted\\
[[END_OF_CHUNK]]&[[NULL]]\\
[[STRING]]&A string with no newlines\\
[[LITERAL]]${}^*$&A string with no newlines, to get no further formatting\\
[[USE_CHUNK]]&The name of the chunk referred to\\
[[NEWLINE]]&[[NULL]]\\
[[DEFINITION]]${}^*$&An identifier whose definition is in this chunk.\\
[[WHATSIT]]&An uninterpreted string passed to the callback function
\end{tabular}
\end{quote}
Items marked with a star ($*$) are legal only when the prettyprinter
is calling back into the application.
The identifier given in [[DEFINITION]] isn't printed, so if the
prettyprinter wants to print it, it has to send it twice.

@

The prettyprinter exports two calls to the application.

<<API for prettyprinter>>=
typedef struct prettyprinter *PrettyPrinter;

extern PrettyPrinter new_prettyprinter(PrettyPrinterCallback callback, void *closure);
extern void prettyprint(PrettyPrinter pp, Parttype type, char *contents);

@

The prettyprinter accepts calls to the [[prettyprint]] procedure.
Any call to [[prettyprint]] may result in one or more callbacks to the
procedure registered when [[new_prettyprinter]] is called.
For any given [[PrettyPrinter]], the application must call
[[prettyprint]] with [[type==END_OF_CHUNK]] exactly once, and that
call must be the last call.

<<header>>=
<<API for prettyprinter>>
@








\section{Implementation}

\subsection{Overview}

The implementation will use a prettyprinter generated by
the Pretzel system. A Pretzel prettyprinter comes in the form of a
class:
 %
\begin{verbatim}
     // abstract base class of prettyprinter:

  #include<iostream>
  #include"attr.h"
  #include"output.h"

  class Pparse {
  public:
    Pparse() {};
    ~Pparse() {};
    virtual int prettyprint(istream*, ostream*) = 0;
    virtual int prettyprint(istream*, Output*) = 0;
  };

     // actual prettyprinter:

  #ifndef PPARSE_NAME
  #define PPARSE_NAME Ppparse
  #endif

  class PPARSE_NAME : public Pparse {
  public:
    PPARSE_NAME();
    ~PPARSE_NAME();
    int prettyprint(istream*, ostream*);
    int prettyprint(istream*, Output*);

    void debug_on();
    void debug_off();
  };
\end{verbatim}
 %
The idea of the implementation is to translate calls to the API
functions into suitable sequences for the Pretzel prettyprinter. This
is not the nicest way of doing it, but it seems the fastest solution
at the moment. (A more elegant way would be to change Pretzel
internals, but that is a different story.)

I have analyzed the code of {\tt pretty.nw} and am using it as the
filter which generates the calls to this prettyprinter. There will be
calls using the [[START_OF_CHUNK]], [[NEWLINE]], [[STRING]],
[[USE_CHUNK]] and [[WHATSIT]] part types. The final call to
[[prettyprint]] is with an [[END_OF_CHUNK]].

Here's an overview over the implementation.

<<*>>=
/* implementation of a noweb prettyprinter using Pretzel */

<<Header files to include>>
<<Handy C routines>>
<<The prettyprinter itself>>
<<The prettyprinter functions>>

@ Of course, we include our own header file, which is {\tt pretty.h}
from the noweb distribution.

<<Header files to include>>=
#include "pretty.h"

@ Because a Pretzel prettyprinter needs to see the entire text before
it can start to prettyprint, we have to wait until an [[END_OF_CHUNK]]
is sent, collecting all the strings and newlines coming in
along the way.  Having done this, we have a view of the entire code
part, including chunk names, etc.~and can send all this to the Pretzel
prettyprinting class (using C++ [[stringstream]]s). Pretzel returns a
long string again which is chopped up into lines and sent back to the
application via the callback function.

There are a few problems here:
 %
\begin{itemize}
\item Chunk uses have to be known to the prettyprinter in order to get
  the formatting (indentation etc.) right. This means that they have
  to be part of the prettyprinting grammar of pretzel.
\item The [[WHATSIT]] parttypes would mess up the prettyprinter if
  they were sent to it because they will inevitably be seen by the
  parser and would mess up parsing! So we sent a [[WHATSIT]] directly
  back to the application if we see it.
\end{itemize}


\subsection{The Actual Prettyprinter}

This the actual prettyprinter. We'll use a [[struct]] to conform with
the API. A [[class]] would be nicer though. Inside the prettyprinter
we have to store the callback function and the [[closure]] parameter,
which is a kind of user data (?). Also, we need a [[stringstream]] object to
collect all the strings coming in and a string to store the name of
the chunk in case we need to do debug output.

<<The prettyprinter itself>>=
struct prettyprinter {
    PrettyPrinterCallback ppcb;
    void* user_data;
    std::stringstream* to_be_pped; /* space for collecting code */
    char* chunk_name; /* name of the chunk being pp'ed */
};
@
<<Header files to include>>=
#include <sstream>

@


\subsection{The Prettyprinter Functions}

To create a new prettyprinter for a chunk, we simply create a new
[[PrettyPrinter]] object and store the interesting stuff in the right
places. (Sorry, this is C++, but we're using [[iostreams]] anyway.)

<<The prettyprinter functions>>=
PrettyPrinter new_prettyprinter( PrettyPrinterCallback callback,
        void* closure)
{
    PrettyPrinter pp = (PrettyPrinter)
            malloc(sizeof(struct prettyprinter));
    pp->ppcb = callback;
    pp->user_data = closure;
    pp->to_be_pped = new std::stringstream;
    return(pp);
}

@
<<Header files to include>>=
#include <stdlib.h>

@ Now for the actual [[prettyprint]] function. Its task is to
prettyprint the incoming strings and send them back to the application
via the callback.

Internally, [[prettyprint]] is a giant switch. It looks at the type of
thing that is coming in and decides what to to.

<<The prettyprinter functions>>=
void prettyprint(PrettyPrinter pp, Parttype type, char *contents)
{
    switch (type) {
        <<Cases of [[prettyprint]] switch>>
    default : cerr << "Oops. There's a parttype I don't understand."
                << endl;
                exit(1);
    }
}

@ Now let's look at the different cases. We know from the start that
the case of [[END_OF_CHUNK]] will be the most complicated, because all
work is done there. So let's leave this until the end. We'll have the
simple cases first.

@

One of the first calls to the prettyprinter will be one with
[[type==START_OF_CHUNK]]. This call doesn nothing but remember the
chunk name. This is okay since typesetting the chunk name is done by
the usual {\tt noweb} back end. String contents have to be copied.

<<Cases of [[prettyprint]] switch>>=
case START_OF_CHUNK : pp->chunk_name = (char*) malloc(strlen(contents)+1);
  strcpy(pp->chunk_name, contents); break;
@

<<cleanup at end of prettyprinting>>=
delete pp->chunk_name;
@

When [[type==STRING]], the incoming string is a bit of code that needs
to be mangled and thus simply appended to the [[stringstream]] within the
prettyprinter.

<<Cases of [[prettyprint]] switch>>=
case STRING : *(pp->to_be_pped) << contents; break;

@ A similar thing is done when a [[NEWLINE]] arrives.

<<Cases of [[prettyprint]] switch>>=
case NEWLINE : *(pp->to_be_pped) << "\n"; break;

@ Now for one of the tricky bits, the [[USE_CHUNK]] case.

Normally, we would call the callback with the chunk name at that
moment, when the prettyprinted text before the chunk name has been
output (via the callback) and before the text following it is output.
Unfortunately, in the current version of Pretzel, we do not have
real control where the chunk name appears in the output and thus don't
have an obvious chance of calling the callback directly here.

We use a trick here: We know that we'll chop up the prettyprinted
output into lines and output these lines one at a time later on. So
we'll put a full line of text referring to the chunk into the
prettyprinted output. This line has to be spotted by the prettyprinter
and left unchanged. When the prettyprinter has done his work, we'll
check all the lines for their starting characters and call the
callback with [[type==USE_CHUNK]] if a line starts with [[@use]].

<<Cases of [[prettyprint]] switch>>=
case USE_CHUNK : *(pp->to_be_pped) << "\n@use " << contents << "\n";
    break;

@

The last case for now is the [[WHATSIT]] case. These things can
be ignored by the pretzel prettyprinter but they will nevertheless
have to be mangled by it. We follow the same plan as above and add the
incoming line to the string that is to be prettyprinted. The
prettyprinter has to spot this line and leave it unchanged.

We assume that these lines have an [[@]] sign at their beginning.

<<Cases of [[prettyprint]] switch>>=
case WHATSIT : *(pp->to_be_pped) << "\n" << contents << "\n";
    break;

@

If an [[END_OF_CHUNK]] is received, we expect it to be the last call
to the current prettyprinter. All code sent to us before has to be
prettyprinted now and output through the callback to the application.
This is where the Pretzel function comes into action.

Say, we have a Pretzel prettyprinter for your favourite programming
language. We need to include its header to be able to use it.

<<Header files to include>>=
#include "Ppparse.h"

@ Ok, now we can outline the case. In general, we'll top off the string
and feed it to the [[prettyprint]] function. This will return stuff on an
[[ostream]], which is replaced again by a [[stringstream]] by us. As most
prettyprinters generated by Pretzel are a little selective concerning
the input that they can handle, we'll check whether or not parsing was
successfull by inspecting the return value of the [[prettyprint]]
function. (This is a to date undocumented feature. The selectiveness
is related to the complexity of writing really good prettyprinting
grammars that can handle everything.) If prettyprinting failed, we'll
output the stuff in normal noweb [[verbatim]] mode (the flag
[[pp_succeeded]] is used to hold this information later)..

Note the initialization values for the two flags: a value of 1 for
[[is_empty]] ensures that the check for emptyness works. A value of 0
for [[pp_succeeded]] ensures that the standard \texttt{noweb} way of
textoutput is used and so no superfluous message of type ``having
problems prettyprinting chunk xyz'' appears.

<<Cases of [[prettyprint]] switch>>=
case END_OF_CHUNK :
  {
    std::ostringstream has_been_pped;
    Ppparse pretzelpp;
    *(pp->to_be_pped) << ends;   /* end up the string */

    char* to_be_pped_string = (char*)pp->to_be_pped->str().c_str();

    int is_empty = 1;   // empty by default
    int pp_succeeded = 0; // didn't succeed by default
    <<check if the chunk is empty and set [[is_empty]] to 0 if not>>

    if (is_empty == 1) {
        // cerr << "String is empty" << endl;
        has_been_pped << "\n" << ends;
    } else {
        <<mangle the chunk contents in [[to_be_pped]] into [[has_been_pped]]>>
    }
    <<Output the resulting text to the application using the callback>>
    delete to_be_pped_string;
  }
  <<cleanup at end of prettyprinting>>
  break;
@

In case we have a truly empty chunk, we must take care not to call
[[prettyprint]] as \pretzel{} cannot deal well with truly empty
chunks. Here, a chunk is considered empty if it contains only newlines
and spaces or if it is really empty (length is 0).

<<check if the chunk is empty and set [[is_empty]] to 0 if not>>=
int l = strlen(to_be_pped_string);
// look if there is a non newline or space
int i = 0;
while (i<l) {
  if ((to_be_pped_string[i] != '\n') && (to_be_pped_string[i] != ' ')) {
    is_empty = 0;
    break;
  }
  i++;
}
@

Here, we're about to do the prettyprinting. Before calling the
[[prettyprint]] function to do this, we have to inspect the
environment for a special flag that tells us whether or not to output
debug information.

<<mangle the chunk contents in [[to_be_pped]] into [[has_been_pped]]>>=
if (getenv("PRETZEL_NOWEB_DEBUG") != NULL)  // stdlib.h already included
  pretzelpp.debug_on();
else
  pretzelpp.debug_off();
@

Now the actual call to the prettyprinter is made. The return value
designated if prettyprinting was successfull (0) or not (1). According
to this value, output of the prettyprinted stuff is different.

<<mangle the chunk contents in [[to_be_pped]] into [[has_been_pped]]>>=
pp_succeeded = pretzelpp.prettyprint(
    pp->to_be_pped, &has_been_pped); /* prettyprint! */
has_been_pped << ends;  /* top off string */
@

In case prettyprinting succeeded, we'll simply chop the stuff in
[[has_been_pped]] into lines and output them to the application as
described in the interface. We'll look at the successfull case first
and then handle the unsuccessfull case.

<<Output the resulting text to the application using the callback>>=
if (pp_succeeded == 0) {
  <<Chop resulting text into lines and send them to application>>
} else if (pp_succeeded == 1) {
  cerr << "having problems prettyprinting chunk <<"
       << (char*) pp->chunk_name
       << // leave line break so noweb doesn't spot a module
          ">>, using verbatim noweb output"
       << endl;
  <<Output the text as normal [[noweb]] output to the application>>
} else {
  cerr << "noweb prettyrinter internal error: illegal yyparse return value"
       << ends;
}
@

The text which comes from the prettyprinter will probably not look
faintly as nice as the original code: Special macros will be added (in
the \LaTeX{} case), indentation will have changed, identifiers may
have swapped places. This is why we'll send the individual lines as
[[LITERAL]] lines to the application. It will ensure that the line
will not get mangled further by {\tt noweb} back ends.

If we encounter a line beginning with [[@]], this is a line to be sent
to the application directly. Assuming that the above lines haven't
been changed by the prettyprinter, this case will cover the
[[USE_CHUNK]] case as well as all other cases of {\tt noweb} keywords.

We'll encircle the output with the \LaTeX{} {\tt ppcode} environment
to get the output right. We also use [[strsep]] instead of [[strtok]]
because the latter can't handle empty fields (see entry in the Pretzel
diary of Dec 16, 1998). However, [[strsep]] isn't as portable as
[[strtok]] (reported by Denis Roegel), and so we'll use our private
version of [[strsep]] called [[LYstrsep]]. It is borrowed from the
latest distribution of Lynx which suffered from the same problem. The
variable [[lookahead]] is introduced for reasons explained below.

<<Chop resulting text into lines and send them to application>>=
char* entire_pped_code = (char*)has_been_pped.str().c_str();
(pp->ppcb)(pp->user_data, LITERAL, "\\begin{ppcode}");
char* line = LYstrsep(&entire_pped_code, "\n");
if (line != NULL) {
  char* lookahead = LYstrsep(&entire_pped_code, "\n");
  do {  
    // cerr << "*** got line : " << line << endl;
    if (strlen(line) == 0) {
      <<Send a newline if necessary>>
    } else {
      if (*line == '@') {
        (pp->ppcb)(pp->user_data, WHATSIT, line);
      } else {
        (pp->ppcb)(pp->user_data, LITERAL, line);
        <<Send a newline if necessary>>
      }
    }
    line = lookahead;
    lookahead = LYstrsep( &entire_pped_code, "\n" );
  } while ( line != NULL );
}
(pp->ppcb)(pp->user_data, LITERAL, "\\end{ppcode}");
@ 
There's a kreeping bug in the code above. Look at the following example:

Assume that the prettyprinter gets something like this to eat
(newlines written explicitly):
 %
\begin{verbatim}
  while i<0 do\n
      \n
  @use Do foo\n
  \n
  if bla;\n
\end{verbatim}
 %
The corresponding input could have looked something like this:
 %
\begin{verbatim}
  while i<0 do
      <Do foo>
  if bla;
\end{verbatim}
 %
On the left you see how the loop defined above will output the code. On
the right you see how it should be:
 %
\begin{verbatim}
  @literal while i<0 do           @literal while i<0 do 
  @nl                             @nl
  @literal [4 spaces]             @literal [4 spaces]
  @nl                             @use Do foo
  @use Do foo                     @nl
  @nl                             @literal if bla;
  @literal if bla;                @nl
  @nl
\end{verbatim}
 %
The difference is the spurious [[@nl]]. It is the result from having
to distinguish ``normal'' newlines from newlines in fron of chunk uses.
Possible cure: send newline only if \emph{no} chunk follows. A chunk
follows if the next line begins with [["@use "]] (this is why we need
the [[lookahead]]).

<<Send a newline if necessary>>=
// printf("about to strlen [%s][%s]!!!!!!!!\n", line, lookahead);
if ((lookahead != NULL) && 
    ((strlen(lookahead) >= 5) && 
    (strncmp(lookahead, "@use ", 5) == 0))) {
  /* don't emit newline */
} else {
  (pp->ppcb)(pp->user_data, NEWLINE, NULL);
}
@ 


@ Finally, it's our responsibility to cleanup the output [[stringstream]],
because we've called the [[str()]] function on it.

<<Chop resulting text into lines and send them to application>>=
delete entire_pped_code;
@

Now for the unsuccessfull case of prettyprinting. In this case we
don't need to use the stuff in [[has_been_pped]] since it's contents
are meaningless. The value of [[to_be_pped]] should be output just as
it would have been without Pretzel. Note that none of the other cases
in the [[prettyprint]] switch calls the callback function and so
nothing wrong has happened up to now. We simply pass everything
through as if nothing had happened.

Note that normal lines are passed as [[STRING]]s to the application
and not as [[LITERAL]]s.

<<Output the text as normal [[noweb]] output to the application>>=
char* to_be_pped_string = (char*)pp->to_be_pped->str().c_str();
(pp->ppcb)(pp->user_data, LITERAL, "\\pretzelnwcodeimitation{}");
char* line = LYstrsep(&to_be_pped_string, "\n");
do {
  if (strlen(line) == 0) {
      (pp->ppcb)(pp->user_data, NEWLINE, NULL);
      continue;
  }
  if (*line == '@') {
      (pp->ppcb)(pp->user_data, WHATSIT, line);
  } else {
      (pp->ppcb)(pp->user_data, STRING, line);
      (pp->ppcb)(pp->user_data, NEWLINE, NULL);
  }
} while ( (line = LYstrsep( &to_be_pped_string, "\n" )) != NULL );
@

Finally, here's the [[LYstrsep]] routine borrowed from the Lynx
distribution 2.8.2 at \texttt{http://www.slcc.edu/lynx/current/}) (the
filenames were \texttt{src/LYStrings.h} and \texttt{src/LYStrings.c}).

<<Handy C routines>>=
static char *LYstrsep(char **stringp, const char *delim)
{
    char *tmp, *out;
    if (!stringp || !*stringp)          /* nothing to do? */
        return 0;                       /* then don't fall on our faces */
    if (!**stringp) {                   /* empty string: */
        *stringp = 0;                   /* let caller see he's done; */
        return 0;                       /* no tokens in an empty string */
    }
    out = *stringp;                     /* save the start of the string */
    tmp = strpbrk(*stringp, delim);
    if (tmp) {
        *tmp = '\0';                    /* terminate the substring with \0 */
        *stringp = ++tmp;               /* point at the next substring */
    }
    else *stringp = 0;                  /* this was the last substring: */
                                        /* let caller see he's done */
    return out;
}
@ 
As there's still a reference to [[strpbrk]] we still must include
the [[string.h]] header file.

<<Header files to include>>=
#include <string.h>
@

\end{document}


%
% $Log: nowebpretzelpp.nw,v $
% Revision 1.8  1999/01/19 10:59:54  felix
% strsep is not a good candidate to use since it is really not very
% portable (Denis B. Roegel reports). Used LYstrsep, an implementation
% of strsep from the Lynx distribution 2.8.2 (thanks for Denis for
% pointing this out).
%
% Revision 1.7  1998/12/16 11:15:11  felix
% Fixed the strtok bug: strtok can't manage empty sequences, using strsep
% instead. Also managed to emit newlines only when necessary.
%
% Revision 1.6  1998/02/24 15:18:56  gaertner
% Handles empty chunks better now and emits the
% \pretzelnwfake... macro before standard noweb verbatim chunks.
%
% Revision 1.5  1998/02/23 13:10:30  gaertner
% Cleaned up code in END_OF_CHUNK case. Outputs debug info now
% in case it can't prettyprint. For this purpose the first call
% to prettyprint with START_OF_CHUNK is remembered.
%
% Revision 1.4  1998/02/23 09:38:23  gaertner
% Changed document title.
%
% Revision 1.3  1996/12/16 19:06:29  gaertner
% release 2.0
%
% Revision 1.2  1996/12/05 09:09:54  gaertner
% Can handle empty chunks now. Removed debug output.
%
% Revision 1.1  1996/12/04 17:09:17  gaertner
% Initial revision
%
%
%
