Parser 5: PEG grammar without memoization

This is loosely based on Daan Leijen and Erik Meijers’ 2001 paper [1]. I say loosely as it lacks all the important elements demonstrated in the paper — efficiency, useful error messages, etc. — but it is a monadic parser.

(Why do you want a monadic parser? This author explains better than I)

Here, the type of a parser is ‘string -> maybe (a, string)’, where ‘a’ is your parse tree, and the result string is the remaining input. If the parse fails, None is returned instead of a tuple.

1
2
3
4
5
6
7
8
9
10
11
12
13

# simplified monadic (PEG) parser. no memoization, some backtracking.
#  parser :: str -> maybe (value, str)
 
def ret (value):
    return lambda s: (value, s)
 
def bind (p, *fs):
    def parse (s):
        res = p(s)
        for f in fs:
            res = res and f(res[0])(res[1])
        return res
    return parse

These are the monad operators return and bind. ‘Ret(v)’ produces a parser that consumes and empty string, producing the parse tree ‘v’, but the juice is in ‘bind’.

‘Bind(p, f)’ glues together a parser and a function. A parser ‘p’ consumes some input and produces a parse tree ‘v’. This value ‘v’ is then passed to a function ‘f’, returning a new parser to consume the remaining input. That is, ‘f’ chooses, based on the parse tree so far, which language to use to interpret the rest of the input.

This is extremely powerful, and permits monadic parsers the ability to recognize classes of context sensitive grammars, such as parsing XML, or loading new languages on the fly as you parse (Perl6, anyone?) or perhaps backtracking to recover if an evaluation of the parse tree fail (Perl5, anyone?).

It can also inhibit various parser optimizations if you’re not careful.

15
16
17
18
19
20
21
22
23

never = lambda s: False
 
def alt (*ps):
    def parse (s):
        for p in ps:
            res = p(s)
            if res: return res
        return False
    return parse

‘never’ is the MonadZero value, a sort of additive identity for the ‘alt’ MonadPlus operator. ‘alt’ produces a parser that will recognize any of the languages passed to it. ‘never’ recognizes nothing, so e.g. ‘alt(p, never)’ is equivalent to ‘p’ in the same way as ‘bind(p, ret)’ is equivalent to ‘p’.

You can ignore these equivalences for now — they’re useful though when it comes time to optimize grammars, but that’s not on today’s agenda.

From here on out, we have a handful of various other operators:

25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

empty = ret(None)
 
def then (p, *ps):
    return bind(p, lambda x: bind(then(*ps), lambda rest: ret([x]+rest)) if ps else ret([x]))
 
def repeat (p, res=[]):
    return alt( bind(p, lambda v: repeat(p, res+[v])),
                ret(res) )
def repeat1 (p):
    return bind(p, lambda v: repeat(p, [v]))
 
def pred (f):
    def parse (s):
        if s and f(s[0]): return (s[0], s[1:])
        else: return False
    return parse
 
def option (p):
    return alt(p, empty)
 
def repn (p, n):
    return then(*([p]*n))
 
def char (ch):
    return pred(lambda c: c==ch)
 
def literal (s):
    return then(*(char(c) for c in s))
 
def delay (fn):
    return bind(empty, lambda _:fn())
 
end = lambda s: False if s else (None, s)

‘Empty’ recognizes the empty string, ‘then’ chains parsers in sequence, ‘repeat’ is Klein star, ‘repeat1′ likewise plus. ‘option’ recognizes one or zero of a language, while ‘repn(p,n)’ recognizes exactly n occurrences of p.

‘Pred(fn)’ consumes one character ‘c’ for which ‘fn(c)’ is true (e.g. using str.isalpha). ‘char(c)’ recognizes the character ‘c’, and ‘literal(s)’ the sequence of characters in the string ‘s’. ‘delay(lambda:p)’ recognizes ‘p’ — and is merely an artifact of strict evaluation, as ‘p’ might not yet be defined. Finally, ‘end’ recognizes only the end of the input string; not very useful in today’s example, but it can be handy in parsers that perform lookahead.

You may not have noticed as it whizzed by, but this parser doesn’t perform full backtracking. The ‘alt’ operator attempts to parse using the first language, eagerly, and if it succeeds, returns immediately. If a subsequent parse operations should fail, no backtracking is performed to consider the alternate path. As such, this parser combinator library might be classified as a parsing expression grammar. Contrast this with context free grammars, where both sides of the alternation are considered equal for backtracking / lookahead purposes.

Size: ~60 lines

Parser 7: CSG with partial memoization

This next parser adds a couple tweaks. First, the type of the parser has changed to ‘string -> lcons of (a, string)’, where lcons is a kind of lazy list.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

class lcons (object):
    def __init__ (self, iter): self.iter = iter; self.value = None
    def force (self):
        if self.iter:
            try:
                self.value = (self.iter.next(), lcons(self.iter))
            except StopIteration:
                self.value = None
            self.iter = None
        return self.value
    def empty(self): return self.force() == None
    def head (self): return self.force()[0]
    def tail (self): return self.force()[1]
    def __iter__ (self):
        while not self.empty():
            yield self.head()
            self = self.tail()
 
def memoize(fn, _memo=dict()):
    def replc (s):
        k = (fn, s)
        if k not in _memo:
            _memo[k] = lcons(iter(fn(s)))
        return _memo[k]
    return replc

‘lcons’ only role is ensure that an iteration is computed only once; otherwise, we’ll be using Python iterators. Speaking of memoization, we’ll also memoize of our monad parser so that we don’t recompute them on backtrack. This … will consume quite a bit of memory, but will reduce the backtracking costs in some key places. You can disable this with otherwise no change in behavior.

27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

def ret (value):
    return lambda s: [(value, s)]
 
def bind (p, f, *fs):
    @memoize
    def parse (s):
        for res1 in p(s):
            for res2 in f(res1[0])(res1[1]):
                yield res2
    if fs: return bind(parse, *fs)
    return parse
 
def alt (*ps):
    @memoize
    def parse (s):
        for p in ps:
            for res in p(s):
                yield res
    return parse
 
def then (p, *ps):
    more = then(*ps) if ps else None
    return bind(p, lambda x: bind(more, lambda rest: ret([x]+rest)) if ps else ret([x]))

The monad parser combinators have received a facelift. ‘bind(p,f)’ now evaluates each possible parse of ‘p’, passing the result to ‘f’ and then enumerating the results from that parser, lazily yielding them to be memoized by ‘lcons’.

‘Alt’ has similarly been updated — each possible parse result from each parser is yielded in turn.

‘Then’ gets a minor performance improvement. As we’re memoizing parse results, we get a boost by reusing the same tail parser each time.

‘Ret’, ‘never’, ‘end’, ‘char’, and ‘pred’ are each updated to return arrays of results, but are otherwise unchanged. All the other parser combinators remain as before.

As a result of these changes, parser7 now supports nearly the class of context free grammars (CFG), save its inability to handle left recursion (perhaps the techniques in [2] could fix that). It can even handle ambiguous grammars, returning a parse forest instead of a parse tree. Of course, nothing is free — parser7 is slower than parser5, and most useful CFGs can be rewritten as PEGs with some effort.

Above and beyond CFGs, this parser continues to provide monadic bind, so can continue to parse a number of useful languages from the class of context sensitive languages. For example, the mini-xml parser in the samples bellow runs great under both libraries.

It’s a memory hog, but it’s also ~90 lines of vanilla Python.

Full source, samples, references

I’ve put the parser combinators and some samples up on codepad.org so you can get a feel for what the output looks like. It’s not pretty, but it’s pretty well structured:

• parser5: http://codepad.org/P5l2l6dm
• parser7: http://codepad.org/pmpqp1wI

[1] Parsec: Direct Style Monadic Parser Combinators For The Real World. Daan Leijen, Erik Meijer (2001).

[2] Packrat Parsers Can Support Left Recursion. Alessandro Warth, James R. Douglass, Todd Millstein (2007).

• “Yacc is dead” (ltu discussion)
• “Yacc is not dead“
• “Yacc is dead: and update“

Like Schrodinger’s cat, Yacc seems to be indeterminately alive or dead (though the last article conclusively opened the box for me).

Concepts are supposed to bring soundness in through constrained templates. Concepts look awfully a lot like type classes; they export functions and types, and are parameterized, and act as constraints on generic functions and other concepts. I checked the draft specification [2], and it even seems to permit parameterizing concepts on type constructors, just like Haskell! (er, C++ calls them class templates, not type constructors. to-may-to, to-mah-to)

But I worried I may be mistaken about concepts, as I’ve searched through google and literature and have yet to find a single example in literature demonstrating the use of template template concepts.

In case you’re curious what this might look like, here’s an educated guess:

concept Monad<template <> class m> {
  template<typename T, typename U>
  m<U> mbind(m<T>, function<m<U>(T)>);
 
  template<class T>
  m<T> mreturn(T);
};
 
template <template <typename> class m,
          class T,
          class U,
          class Iter>
requires M<m>
requires InputIterator<Iter, U>
m<T> foldM(Iter begin, Iter end, T i, function<m<T>(T, U)> f)
{
    if(begin == end)
        return mreturn(i);
    else
        return mbind(f(i, *begin), [=](T result){
            Iter next = begin;
            ++next;
            return foldM(next, end, result, f);
        });
}

Have template template concepts been covered thoroughly somewhere, and I’ve just missed it?

• [1] “C++ templates/traits versus Haskell typeclasses” (2005), by Sunil Kothari, Martin Sulzmann.
• [2] “Proposed Wording for Concepts (Revision 5)” (2008).
• [3] “Trip Report: Exit Concepts, Final ISO C++ Draft in ~18 Months” (2009), Herb Sutter.
• [4] “ConceptClang: An Implementation of C++ Concepts in Clang” [pdf]

Online REPs and REPLs

Today there’s 61 different languages on that list. That many, there’s gotta be at least one that strikes your fancy.

Best batch execution site: ideone.com with 50 unique languages.

Best interactive REPL site: repl.it with 16 unique languages.

There was a particular line by the annotator that continues to amuse me:

Dijkstra seems to imply that iterative looping (inductive) statements are intellectually harder to grasp than recursion, which is the kind of thing only a mathematician would say.

Among the ways you can bisect computer science, is whether this statement insults or compliments.

CL-PocketMod draft 1

For reference, the prior quickrefs are here: [link]

But then turns around and cries out “one way to do it“, leaving the programmer perplexed as to how Guido van Rossum thought we should do things. For example, Guido hates tail calls, so recursion isn’t the one way to do it that he picked (note that his blog post and followups contain a large number of factual errors; read it as an opinion piece only).

During these fits, Python suffers itself to be a bondage and discipline language.

Apparently there is hope. One bit that causes me particular pain is that nested functions cannot rebind variables in the outer scope; only read from them. A case in point:

def f():
  x = 1
  def doubleIt():
    x *= 2 # local variable 'x' referenced before assignment
  doubleIt(); doubleIt();
  return x;

However, I was reading a piece on Factor, and it mentions that this restriction is lifted in Python 3.0. I’m still on 2.6 (the only differences I had been aware of were the somewhat arbitrary swapping of the ‘/’ and ” operators, and the insulting removal of map, filter, and reduce from Python 3.0). However, fixing this scoping rule, even if an extra keyword is needed, sure would be convenient for me.

Actually, there are a lot of improvements:

• Various APIs returns views instead of mutable list (copies).
• sorted is now built in. (I’ve had to write that myself so many times…)
• Apparently map and filter are still here (though he still went and put reduce all the way off in functools. I guess you can’t have it all -_-)
• Set literals, yay!

Now, if Guido would be so kind to add proper statement support to lambdas, I’ll switch ^_^. Oh, but Guido hates lambdas. Oh well, next time maybe.

– signed an ambivalent Python user

They reported that if they compiled code which calculated: 1130149156 >> -05701653 it generated different results on 32bit and 64bit operating systems. On 32bit machines it reported 0 but on 64bit machines, it reported 0x21a.

This is one of various areas of “undefined behavior” for which you can ask 2 engineers what it should do and get 3 answers, at least one being that “but I know there must be one!” I think I know where at least 2 of the answers are coming from …

The Hardware shift

On x64, Larry mentions that the sar instruction is used to carry out the operations. This instruction is not new to the x86 architecture, and in that architecture line’s early days you did not waste die space. So what’s cheap?

Enter the barrel shifter. It’s cheap, it’s small, you pick the maximum power-of-2 shift you want, and that’s what you pay for. For a 16-bit shift, you can support shifts of 0 to 15 and pay for 4 stages. If you want to shift out all 16, you might think “add another stage” — but it gets worse! It only takes rotation as input. If you need to identify greater shifts than 31, then you’re toast.

No, what you consider doing is detect shifts > 15, and add a an extra stage to mask away the entire output. But wait, it’s 1972, and you don’t have either the die space nor the extra cycles to waste on a conditional masking stage (masking to 4 bits of shift input is free — you just don’t connect the other 12 input bits to anything). You just tell the programmer to never shift too much, use the bottom most bits, and ignore the rest. (1 >> 128) == 1.

In time the processor is expanded to support 32-bit quantities and 64-bit quantities, but each time you need to remain compatible, so the sar instruction continues to mask away the upper bits.

The Programming Language

I’m gonna gloss over the whole “undefined behavior” thing. Enough others have discussed it. Even languages or implementations which aim to minimize it will occasionally miss edge cases or fall short.

And C takes a very brutal route of making a great many things “undefined behavior” in the name of permitting performance shortcuts. For instance, signed arithmetic overflow yields undefined behavior. This might seem strange now, but at the time 2′s compliment arithmetic was not common. Choosing 2′s compliment wrap-around would have introduced perf penalties on 1′s compliment architectures as they did not generally include native 2′s compliment instructions, and vice-versa. A modern spin on this are special purpose DSP chips that default to saturating arithmetic.

In any case, I imagine a similar case stands for shifting. One architecture may prefer precise shifting, but x86 prefers to take shifts in modulo. Rather than specify one behavior and penalize compilation on all other hardware, the C language says instead “this area off limits — if you want a particular slow-but-precise behavior, implement it yourself”

The Software Shift

The 32-bit x86 CRT routine mentioned was _allshr (the extra leading underscore comes from cdecl calling convention on x86), and you can find its source in your Visual C++ product installation, under %programfiles%/Microsoft Visual Studio 10.0/vc/crt/src/intel/llshr.asm. It leads off:

;
; Handle shifts of 64 bits or more (if shifting 64 bits or more, the result
; depends only on the high order bit of edx).
;
        cmp     cl,64
        jae     short RETSIGN

Interesting — for large shifts, it bails early. Following it is another branch, switching between different implementations for 0-31 and 32-63 bit. In any case, the resulting 3 code paths are short, with the cl >= 64 path being the shortest of all.

Now, the designer could have considered masking instead of comparing — but for random shift amounts, that would have been a longer code sequence to execute*. And by most reasonable interpretations, returning the sign bit is the right answer.

So there you have it

Each implementation is most efficient and reasonable, when and where it was designed, and both are permitted by a language standard that valued efficiency above most else. And in that respect, both implementations were perfect … at least, they were when originally written.

* yes, the conditional branch is probably a bigger cost than the savings… today. While I don’t know when this code was written, it mentions PROC NEAR, leading me to think that perhaps it comes from a time before superscalar architectures, where branches were cheap(er) and arithmetic was costly. You might be want to change it now, but that decision opens up back-compat questions.

Wait what. Python compiles? That is correct. CPython and PyPy (the implementations worth caring about currently) are in fact creating a code object from the string you pass to exec or eval before executing it. And that’s just one of the things many people don’t know about the exec statement.

It doesn’t always have to be about theory, nor should it be. I might recommend this article for programmers already versed in Python, to read side by side or leading up to SICP’s “Metalinguistic Abstractions” chapter.

I applied for SDK access, but heard nothing back. So instead, I’m using the built in “experimental” web browser. It has canvas support, so I’m golden.

As you might have noticed, I couldn’t resist the urge to design a vector font for the task; please be gentle, I know the font leaves a lot to be desired, it’s a work in progress. The Kindle browser didn’t seem to have font support anyway, while public domain, I didn’t particularly enjoy the Hershey fonts. The Hershey fonts are nice, but they’re unsuitable for programming.